DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on March 17, 2026 has been entered.
Response to Arguments
Applicant’s arguments, see page 7, filed March 17, 2026, with respect to 35 USC 112(b) have been fully considered and are persuasive. The rejection of the claims under 35 USC 112(b) has been withdrawn.
Applicant’s arguments with respect to claims 1-5, 7-19 and 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
The amended language is considered to be disclosed in light of Lee and the independent claims are considered obvious when Russo is considered in view of Lee. Please see 35 UCS 103 Rejections section below on how the claims are considered to be obvious in view of the prior art references..
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 7, 8, 10, 11, 19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Russo (U.S. Publication No. 2007/0196033 A1, hereinafter referred to as “Russo”) in view of Lee et al. (U.S. Patent No. 9,335,835 B2, hereinafter referred to as “Lee”).
Regarding claim 1, Russo discloses an information communication terminal device that is communicably connectable to a search server, comprising: (e.g., figure 1 and paragraphs [0015]-[0018])
a user interface unit that includes a touch panel; (tablet that has interface with touch pad)(e.g., figure 2 and paragraphs [0022], [0023] and [0026])
a search query acquisition unit that acquires a search query; (“FIGS. 8-11 show various states of an illustrative search interface. In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T". There are other variations on how to interpret the criterion "T."For instance, computer 100 may search for those photos/portions that have the letter "T" anywhere in an associated label (not just at the beginning of a word).”)(figures 8-11 and paragraphs [0038]-[0039])
a recognition unit that performs first recognition processing based on line drawing data handwritten at a location a screen that represents a virtual space on the touch panel; (“In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T".”)(e.g., figure 8 and paragraph [0038])
a search result acquisition unit that acquires search results containing multiple search result images transmitted from the search server in response to the search query; and (“Computer 100 may search for photos and/or photo portions in any of a number of ways. For example, as previously mentioned, computer 100 may automatically update and maintain a central index of labels (or label portions) as the user inputs the labels. The index, which may be separate from the files embodying the photos themselves, may indicate an association between each label and each photo and/or portion thereof, and may be searchable by letter, by word, by phrase, and/or by symbol contained in the labels. Where such an index is used, computer 100 may search by simply referring to the existing index. Or, computer 100 may search information that is part of or linked to the photo files. For example, computer 100 may search metadata that is part of each photo file.” “computer 100 may automatically update and maintain a central index of labels (or label portions) as the user inputs the labels. The index, which may be separate from the files embodying the photos themselves, may indicate an association between each label and each photo and/or portion thereof, and may be searchable by letter, by word, by phrase, and/or by symbol contained in the labels. Where such an index is used, computer 100 may search by simply referring to the existing index. Or, computer 100 may search information that is part of or linked to the photo files. For example, computer 100 may search metadata that is part of each photo file.”)(e.g., figure 1 and paragraphs [0016], [0024] and [0039])
a display control unit that performs control for displaying each of the multiple search result images contained in the search results on the screen, (“The user may institute a search as described above, however instead of or in addition to providing a simultaneous set of search results such as in FIGS. 8-11, a sequential automated slideshow may be used to display the search results.”)(e.g., figures 8-13 and paragraphs [0047]-[0050]) wherein the display control unit performs control such that the line drawing data is converted into an icon including text representing the line drawing data that is displayed on the screen, and each of the multiple search result images is displayed in animated form in sequence with a predetermined time difference, each of multiple search result images appearing at a predetermined rate within a predetermined period of time from a first position determined with respect to the search result image within the screen on the touch panel. (“The user may request this by, for example, selecting a slideshow mode. Alternatively, the slideshow mode may already be selected. In response to selecting the slideshow mode and/or in response to the search results 802, 803 being determined, a slideshow is presented that sequentially displays thumbnails 802 and 803 (or full versions of the photos or photo portions represented by thumbnails 802, 803).” “one or more various transitions (for instance, cross-fading, swiping, etc.) may be used between each displayed search result.”)(e.g., paragraphs [0049]-[0050]).
However, Russo does not appear to specifically disclose the line drawing data is converted into an icon including text representing the line drawing data that is displayed on the screen, and
On the other hand, Lee, which relates to an interface that includes handwriting recognition (e.g., abstract), does disclose the line drawing data is converted into an icon including text representing the line drawing data that is displayed on the screen, and (“FIG. 11 is a diagram illustrating an example of the handwritten input data. The user inputs desired contents into the touch screen 190 by using the stylus pen 168, that is, handwritten data 1110 having contents of “weather in Seoul” or “Seoul weather” in the present embodiment. In the present embodiment, since an application window 710e occupies most of the touch screen, the user inputs the handwritten data 1110 into the application window 710e.” “FIG. 12 is a diagram illustrating an example of a method of processing converted text data. The application displays converted text data 1210 on an application window 710f, searches for weather of Seoul by inputting the converted text data 1210 as a search word, and displays a search result 1220 on the application window 710f. At this time, the application can search for the weather of Seoul also by inputting the converted text data 1210 and a current position (e.g., Seogyo-dong) of the terminal 100.”)(e.g., col 13 lines 47-54 and col 14 lines 12-21).
Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose that the line drawing data is converted into an icon including text representing the line drawing data that is displayed on the screen. On the other hand, Lee, which also relates to performing handwriting recognition (e.g., abstract), discloses that based on the user’s input, an icon, along with text can be generated and presented to the user based on the user’s handwritten input. E.g., figures 11 and 12. This provides an enhanced manner for user’s to enter information and to see their intention is being interpreted for receiving subsequent results. E.g., col 1 lines 40-46. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the generation of icon/text of the user’s handwritten notes as disclosed in Lee to Russo to provide confirmation to the user that the subsequent results match the user’s intentions.
Regarding claim 2, Russo in view of Lee discloses the information communication terminal device according to claim 1. Russo and Lee further discloses wherein the search query acquisition unit acquires a text recognized by the recognition unit as the search query. (“In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T".”)(Russo: e.g., figure 8 and paragraph [0038])(Lee: e.g., figure 12).
Regarding claim 3, Russo in view of Lee discloses the information communication terminal device according to claim 2. Russo further discloses wherein the recognition unit starts performing the first recognition processing in response to a user's manipulation action to the line drawing data on the screen of the touch panel. (“For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T". There are other variations on how to interpret the criterion "T."For instance, computer 100 may search for those photos/portions that have the letter "T" anywhere in an associated label (not just at the beginning of a word). Or, for instance, computer 100 may search for those photos/portions that have an associated label beginning with the letter "T." In addition, searches may be literal such that, for example, only exact matches are provided in the search results. Alternatively, searches may be fuzzy searches. More generally, computer 100 searches for those photos and/or portions thereof having an associated label meeting the input criterion.”)(e.g., paragraph [0038]).
Regarding claim 7, Russo in view of Lee discloses the information communication terminal device according to claim 1. Russo further discloses wherein the mode of the display in animated form is at least one of: a mode in which the search result image is completed visually at a predetermined period of time; a mode in which the search result image is visually expressed in perspective; and a mode in which the search result image is visually enhanced. (“Each search result may be presented in any manner desired, such as full-screen photos/portions or smaller versions thereof. In addition, one or more various transitions (for instance, cross-fading, swiping, etc.) may be used between each displayed search result.” – cross-fading is considered to be completed visually at a predetermined period of time)(e.g., paragraph [0050]).
Regarding claim 8, Russo in view of Lee discloses the information communication terminal device according to claim 1. Russo further discloses wherein the display control unit determines the first position on the screen for each of the multiple search result images and performs control such that each search result image is displayed at the determined first position. (e.g., figures 8-13 and paragraphs [0049]-[0050]).
Regarding claim 10, Russo in view of Lee discloses the information communication terminal device according to claim 8. Russo further discloses wherein the display control unit randomly determines the first position for each search result image. (“The slideshow may further display the search results in a particular order (such as by relevance or date modified) or randomly.”)(e.g., paragraph [0048])
Regarding claim 11, Russo in view of Lee discloses the information communication terminal device according to claim 1. Russo further discloses wherein the display control unit performs control of the display in animated form such that the displayed search result image disappears at a predetermined rate within a predetermined period of time after a predetermined period of time has elapsed. (“Each search result may be presented in any manner desired, such as full-screen photos/portions or smaller versions thereof. In addition, one or more various transitions (for instance, cross-fading, swiping, etc.) may be used between each displayed search result.” – cross fading is considered to show disappears at a predetermined rate within a predetermined )(e.g., paragraph [0050]).
Regarding claim 19, Russo discloses a method of displaying search results by an information communication terminal device communicably connectable to a search server, the method comprising: (e.g., figure 1 and paragraphs [0015]-[0018])
acquiring a search query from a user's input as a line drawing data handwritten at a location on a screen that represents a virtual space on a touch panel of the information communication terminal device; (“FIGS. 8-11 show various states of an illustrative search interface. In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T". There are other variations on how to interpret the criterion "T."For instance, computer 100 may search for those photos/portions that have the letter "T" anywhere in an associated label (not just at the beginning of a word).” tablet that has interface with touch pad.)(figures 2 and 8-11 and paragraphs [0022], [0023], [0026], [0038]-[0039])
performing first recognition processing based on the line drawing data; (“In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T".”)(e.g., figure 8 and paragraph [0038])
in response to the search query, acquiring search results containing multiple search result images transmitted from the search server; and (“Computer 100 may search for photos and/or photo portions in any of a number of ways. For example, as previously mentioned, computer 100 may automatically update and maintain a central index of labels (or label portions) as the user inputs the labels. The index, which may be separate from the files embodying the photos themselves, may indicate an association between each label and each photo and/or portion thereof, and may be searchable by letter, by word, by phrase, and/or by symbol contained in the labels. Where such an index is used, computer 100 may search by simply referring to the existing index. Or, computer 100 may search information that is part of or linked to the photo files. For example, computer 100 may search metadata that is part of each photo file.” “computer 100 may automatically update and maintain a central index of labels (or label portions) as the user inputs the labels. The index, which may be separate from the files embodying the photos themselves, may indicate an association between each label and each photo and/or portion thereof, and may be searchable by letter, by word, by phrase, and/or by symbol contained in the labels. Where such an index is used, computer 100 may search by simply referring to the existing index. Or, computer 100 may search information that is part of or linked to the photo files. For example, computer 100 may search metadata that is part of each photo file.”)(e.g., figure 1 and paragraphs [0016], [0024] and [0039])
performing control for displaying each of the multiple search result images contained in the search results on the screen, (“The user may institute a search as described above, however instead of or in addition to providing a simultaneous set of search results such as in FIGS. 8-11, a sequential automated slideshow may be used to display the search results.”)(e.g., figures 8-13 and paragraphs [0047]-[0050])
wherein performing control includes performing control to convert the line drawing data into an icon including text representing the line drawing data and display the icon including the text on the screen, and to display each of the multiple search result images in animated form in sequence with a predetermined time difference, each of multiple search result images appearing at a predetermined rate within a predetermined period of time from a first position determined with respect to the search result image within the screen on the touch panel. (“The user may request this by, for example, selecting a slideshow mode. Alternatively, the slideshow mode may already be selected. In response to selecting the slideshow mode and/or in response to the search results 802, 803 being determined, a slideshow is presented that sequentially displays thumbnails 802 and 803 (or full versions of the photos or photo portions represented by thumbnails 802, 803).” “one or more various transitions (for instance, cross-fading, swiping, etc.) may be used between each displayed search result.”)(e.g., paragraphs [0049]-[0050]).
However, Russo does not appear to specifically disclose to convert the line drawing data into an icon including text representing the line drawing data and display the icon including the text on the screen, and
On the other hand, Lee, which relates to an interface that includes handwriting recognition (e.g., abstract), does disclose to convert the line drawing data into an icon including text representing the line drawing data and display the icon including the text on the screen, and (“FIG. 11 is a diagram illustrating an example of the handwritten input data. The user inputs desired contents into the touch screen 190 by using the stylus pen 168, that is, handwritten data 1110 having contents of “weather in Seoul” or “Seoul weather” in the present embodiment. In the present embodiment, since an application window 710e occupies most of the touch screen, the user inputs the handwritten data 1110 into the application window 710e.” “FIG. 12 is a diagram illustrating an example of a method of processing converted text data. The application displays converted text data 1210 on an application window 710f, searches for weather of Seoul by inputting the converted text data 1210 as a search word, and displays a search result 1220 on the application window 710f. At this time, the application can search for the weather of Seoul also by inputting the converted text data 1210 and a current position (e.g., Seogyo-dong) of the terminal 100.”)(e.g., col 13 lines 47-54 and col 14 lines 12-21).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above.
Regarding claim 21, Russo discloses a product comprising a non-transitory computer-readable medium storing computer program for causing an information communication terminal device to implement a method for displaying search results transmitted from a search server in response to a search query, the method comprising: (e.g., figure 1 and paragraphs [0015]-[0018])
acquiring a search query from a user's input as a line drawing data handwritten at a location on a screen that represents a virtual space on a touch panel of the information communication terminal device; (“FIGS. 8-11 show various states of an illustrative search interface. In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T". There are other variations on how to interpret the criterion "T."For instance, computer 100 may search for those photos/portions that have the letter "T" anywhere in an associated label (not just at the beginning of a word).” tablet that has interface with touch pad.)(figures 2 and 8-11 and paragraphs [0022], [0023], [0026], [0038]-[0039])
performing first recognition processing based on the line drawing data; (“In FIG. 8, an input box 801 is displayed. The user can type or hand write using stylus 166 input that is used as criteria into input box 801. In response to each letter, word, phrase, or symbol being input, computer 100 searches for those photos (and/or portions thereof) having metadata properties in accordance with what has been entered into input box 801. For example, the user has written the letter "T" into input box 801. In response, computer 100 recognizes the hand-written "T" as the text letter "T" and searches for those photos and/or portions thereof that have an associated label having a word starting with the letter "T".”)(e.g., figure 8 and paragraph [0038])
in response to the search query, acquiring search results containing multiple search result images transmitted from the search server; and (“Computer 100 may search for photos and/or photo portions in any of a number of ways. For example, as previously mentioned, computer 100 may automatically update and maintain a central index of labels (or label portions) as the user inputs the labels. The index, which may be separate from the files embodying the photos themselves, may indicate an association between each label and each photo and/or portion thereof, and may be searchable by letter, by word, by phrase, and/or by symbol contained in the labels. Where such an index is used, computer 100 may search by simply referring to the existing index. Or, computer 100 may search information that is part of or linked to the photo files. For example, computer 100 may search metadata that is part of each photo file.” “computer 100 may automatically update and maintain a central index of labels (or label portions) as the user inputs the labels. The index, which may be separate from the files embodying the photos themselves, may indicate an association between each label and each photo and/or portion thereof, and may be searchable by letter, by word, by phrase, and/or by symbol contained in the labels. Where such an index is used, computer 100 may search by simply referring to the existing index. Or, computer 100 may search information that is part of or linked to the photo files. For example, computer 100 may search metadata that is part of each photo file.”)(e.g., figure 1 and paragraphs [0016], [0024] and [0039])
performing control for displaying each of the multiple search result images contained in the search results on the screen, (“The user may institute a search as described above, however instead of or in addition to providing a simultaneous set of search results such as in FIGS. 8-11, a sequential automated slideshow may be used to display the search results.”)(e.g., figures 8-13 and paragraphs [0047]-[0050])
wherein performing control includes performing control to convert the line drawing data into an icon including text representing the line drawing data and display the icon including the text on the screen, and to display each of the multiple search result images in animated form in sequence with a predetermined time difference, each of multiple search result images appearing at a predetermined rate within a predetermined period of time from a first position determined with respect to the search result image within the screen on the touch panel. (“The user may request this by, for example, selecting a slideshow mode. Alternatively, the slideshow mode may already be selected. In response to selecting the slideshow mode and/or in response to the search results 802, 803 being determined, a slideshow is presented that sequentially displays thumbnails 802 and 803 (or full versions of the photos or photo portions represented by thumbnails 802, 803).” “one or more various transitions (for instance, cross-fading, swiping, etc.) may be used between each displayed search result.”)(e.g., paragraphs [0049]-[0050]).
However, Russo does not appear to specifically disclose to convert the line drawing data into an icon including text representing the line drawing data and display the icon including the text on the screen, and
On the other hand, Lee, which relates to an interface that includes handwriting recognition (e.g., abstract), does disclose to convert the line drawing data into an icon including text representing the line drawing data and display the icon including the text on the screen, and (“FIG. 11 is a diagram illustrating an example of the handwritten input data. The user inputs desired contents into the touch screen 190 by using the stylus pen 168, that is, handwritten data 1110 having contents of “weather in Seoul” or “Seoul weather” in the present embodiment. In the present embodiment, since an application window 710e occupies most of the touch screen, the user inputs the handwritten data 1110 into the application window 710e.” “FIG. 12 is a diagram illustrating an example of a method of processing converted text data. The application displays converted text data 1210 on an application window 710f, searches for weather of Seoul by inputting the converted text data 1210 as a search word, and displays a search result 1220 on the application window 710f. At this time, the application can search for the weather of Seoul also by inputting the converted text data 1210 and a current position (e.g., Seogyo-dong) of the terminal 100.”)(e.g., col 13 lines 47-54 and col 14 lines 12-21).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Russo in view of Lee and in further view of Eskolin et al. (U.S. Publication No. 2010/0262616 A1, hereinafter referred to as “Eskolin”).
Regarding claim 4, Russo in view of Lee discloses the information communication terminal device according to claim 3. However, neither reference appears to specifically disclose wherein the first manipulation action is a drawing of a line that encloses at least part of an area indicated by the line drawing data on the screen of the touch panel.
On the other hand, Eskolin, which relates to visual search engine results (title), does disclose wherein the first manipulation action is a drawing of a line that encloses at least part of an area indicated by the line drawing data on the screen of the touch panel. (“FIG. 3E depicts one example of a user interface 356 useful for forming queries to submit to a visual search engine. In the user interface, the user can drag and draw a search box 358 along with adding one or more keywords "dog" as shown in the figure. The search box 358 can be dragged into a desired position within the interface 356 and also be sized as desired. The interface 360 depicts that not only can one search box be defined but two separate search boxes 362, 364 (or more) can be used to form a query. Each box 362, 364 can independently have different sizes, positions, and keywords.”)(e.g., figure 3E and paragraph [0058]).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose that the first manipulation action is a drawing of a line that encloses at least part of an area indicated by the line drawing data on the screen of the touch panel. On the other hand, Eskolin, which also relates to searching images, provides that users can drag and draw a search box along with adding words to perform the searching. This provides an enhanced experience by allowing the user to an alternative way to conduct and initiate the search, which might be a matter of user preference. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention as a matter of design choice as illustrated by Eskolin to incorporate the first manipulation action to have the user draw a line that encloses part of the are indicated by the line drawing data on the screen as an additional mechanism for users to conduct the search within the Russo-Lee combination.
Claims 5, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Russo in view of Lee and in further view of Sternberg et al. (U.S. Patent No. 12,154,021 B1, hereinafter referred to as “Sternberg”).
Regarding claim 5, Russo in view of Lee discloses the information communication terminal device according to claim 2. However, neither reference appears to specifically disclose wherein the recognition unit is configured to perform second recognition processing on image data to generate object information, and the search query acquisition unit acquires the search query based on the generated object information.
On the other hand, Sternberg, which relates to a visual search and content display system (title), does disclose wherein the recognition unit is configured to perform second recognition processing on image data to generate object information, and the search query acquisition unit acquires the search query based on the generated object information. (“When selectable control 212 is selected, computing device(s) 120 determines objects recognized within the image. As described in further detail below, the various content ensemble images displayed in a feed may be processed using an object recognition machine learning model to identify classes of objects represented in the image data for which the machine learning model has been trained to detect. Additionally, in some example embodiments, the object recognition machine learning model generates bounding boxes and/or segmentation masks identifying groupings of pixels in the image data that correspond to a detected object. The object recognition machine learning model generates feature embeddings (e.g., feature vectors) visually representing the objects in a feature space. If the objects pertain to objects present in an online catalog (e.g., for an online retailer), the feature embeddings may be stored in at least one non-transitory computer-readable memory (e.g., non-transitory computer-readable memory 103) in association with label data describing characteristics of the object/product (e.g., object type=sofa, object style=mid-century modem, object_color=purple, object_cost=$649.99, etc.).” “For example, a user can select selectable control 304 on enlarged image 302a to see sofas that are visually similar to the sofa displayed in content ensemble image 305 (as determined by searching for sofa images with a low distance in the feature space from the feature vector representing the sofa displayed in content ensemble image 305). Upon selection of selectable control 304, filters 312 are displayed so that a user can specify attributes that the similar content should include.”)(e.g., col 7 lines 20-40 and 54-61)
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose recognizing object data and performing subsequent searching. On the other hand, Sternberg provides the ability for the system to do this to provide additional content similar to the image of interest This provides an enhanced user experience, because it provides users similar images to selected ones. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the content recognition for further searching as disclosed in Sternberg to the Russo-Lee combination to provide the user with an enhanced experience of expanding search results to include similar features of the selected image.
Regarding claim 17, Russo in view of Lee discloses the information communication terminal device according to claim 1 However, neither reference appears to specifically disclose wherein the display control unit performs control, in response to a user's manipulation action on at least one search result image of no interest from among the multiple search result images on the screen of the touch panel, so as to remove the at least one search result image of no interest from the screen.
On the other hand, Sternberg, which relates to a visual search and content display system (title), does disclose wherein the display control unit performs control, in response to a user's manipulation action on at least one search result image of no interest from among the multiple search result images on the screen of the touch panel, so as to remove the at least one search result image of no interest from the screen. (“Selectable control 202 (sometimes referred to as a “dislike” or “cancel” button) may be effective to remove the image from the feed. In an example, a user may select the selectable control 202 when the user does not like the image (e.g., if the user does like the style of the furniture). Upon selection of selectable control 202, feature data (e.g., a feature vector) visually representing the image (e.g., determined by a CNN and/or other encoder machine learning network) may be determined. In various examples, such feature data may be determined prior to displaying an image on the GUI 102 and may be stored in a database and/or other data structure. Thereafter, computing device(s) 120 may determine an image associated with a feature vector that has a maximum distance (or a distance above a threshold distance and/or a distance within a tolerance of the maximum distance (e.g., 5%, 10% or some other suitable value)) in the feature space from the feature vector representing the disliked image (e.g., the image for which selectable control 202 was selected), from among a set of images satisfying the attributes specified using filters 204.”)(e.g., col 5 line 52 – col 6 line 5).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose the user’s ability to remove content and to update search results based on removed content. On the other hand, Sternberg provides the ability for users to dislike the image, which removes image from screen and also is fed into the system for future results. E.g. col 5 line 52 – col 6 line 5,. This provides an enhanced user experience, because it provides users to remove unwanted images and improve future results by avoiding results that are undesirable to the user. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the dislike feature, which removes unwanted results and filters content to avoid similar results as disclosed in Sternberg to the Russo-Lee combination to provide the user with an enhanced experience of tailoring content to the user’s likes and dislikes.
Regarding claim 18, Russo in view of Lee and in further view of Sternberg discloses the information communication terminal device according to claim 17. Sternberg further discloses wherein the search query acquisition unit acquires, in response to the user’s manipulation action, a new search query based on metadata associated with some search result images devoid of the at least one search result image of no interest from among the multiple search result images. (“Selectable control 202 (sometimes referred to as a “dislike” or “cancel” button) may be effective to remove the image from the feed. In an example, a user may select the selectable control 202 when the user does not like the image (e.g., if the user does like the style of the furniture). Upon selection of selectable control 202, feature data (e.g., a feature vector) visually representing the image (e.g., determined by a CNN and/or other encoder machine learning network) may be determined. In various examples, such feature data may be determined prior to displaying an image on the GUI 102 and may be stored in a database and/or other data structure. Thereafter, computing device(s) 120 may determine an image associated with a feature vector that has a maximum distance (or a distance above a threshold distance and/or a distance within a tolerance of the maximum distance (e.g., 5%, 10% or some other suitable value)) in the feature space from the feature vector representing the disliked image (e.g., the image for which selectable control 202 was selected), from among a set of images satisfying the attributes specified using filters 204.”)(e.g., col 5 line 52 – col 6 line 5).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Russo in view of Lee and in further view of Sun et al. (U.S. Publication No. 2008/0193048 A1, hereinafter referred to as “Sun”).
Regarding claim 9, Russo in view of Lee discloses the information communication terminal device according to 8. However, neither reference appears to specifically disclose wherein the first positions for the respective search result images are not geometrically aligned with each other.
On the other hand, Sun, which relates to picture collage systems and methods (title), does disclose wherein the first positions for the respective search result images are not geometrically aligned with each other. (“A computing device 102 hosts an exemplary collage engine 104. The collage engine 104 receives data representing a digital image collection or image search results 106. The collage engine 104 models the image collection or search results 106 in a Bayesian framework and applies a Markov Chain Monte Carlo (MCMC) technique to create an exemplary picture collage 108 in which the salient region of each image is displayed in an overlay style and balanced with the salient regions of the other images in the collage. Blank space is minimized in the exemplary collage 108 and the (rotational) orientation of each image is diversified in a manner that is pleasing to the human observer.”)(e.g., figures 1, 3 and 6-9 and paragraph [0022]).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose that the result images are not geometrically aligned with each other. On the other hand, Sun, provides that arranging images in a collage minimizes blank space and provides an arrangement of images in a manner that is pleasing to the human observer. E.g., paragraph [0022]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the collage view of Sun to the Russo-Lee combination to minimize blank space and to provide the results in a manner that is pleasing to the human observer.
Claims 12, 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Russo in view of Lee and in further view of Amacker (U.S. Patent No. 8,631,029 A1, hereinafter referred to as “Amacker”).
Regarding claim 12, Russo in view of Lee discloses the information communication terminal device according to claim 1. However, neither reference appears to specifically disclose wherein the display control unit performs control, in response to a user's manipulation action on at least one search result image of interest from among the multiple search result images on the screen of the touch panel, so as to move the at least one search result image of interest to a second position on the screen.
On the other hand, Amacker, which relates to evolutionary content determination and management (title) does disclose wherein the display control unit performs control, in response to a user's manipulation action on at least one search result image of interest from among the multiple search result images on the screen of the touch panel, so as to move the at least one search result image of interest to a second position on the screen. (“A user may view detailed information about any item within a search result set displayed in the page. FIG. 5(a) illustrates an example display 500 wherein selection of an item 554 in search result set by a user causes display of an information window 508. The information window 508 is shown in FIG. 5(b) as a pop-up (or modal) window or panel that provides detailed information about the selected item and a "buy it!" box 510. The particular position of window 508 is exemplary. The window 508 may be displayed in any portion of display 550. Any appropriate information about the selected item can be displayed in window 508, such as a user rating, price, product information (e.g., images, video, etc.), links to related information (e.g., third party reviews, etc.), and the like.”)(e.g., figures 5(a) and 5(b) and col 8 lines 11-24).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose that the user’s second manipulation action causes the image to move positions on the screen. On the other hand, Amacker provides the ability for users to select the image result, and based on the selection, causing the image to move and be shown within another section of the display. E.g., figures 5(a) and 5(b) and col 8 lines 11-24. This provides an enhanced manner to show what is being selected and also provides an effective manner for the display to provide additional details relating to the content and to connect the selected content with the additional details. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the moving or relocating of the selected content in view of Amacker to the Russo-Lee combination to provide the user the ability to select and see additional information related to the result image of interest.
Regarding claim 14, Russo in view of Lee discloses the information communication terminal device according to claim 1. However, neither reference appears to specifically disclose wherein the display control unit performs control, in response to a user's manipulation action on at least one search result image of interest from among the multiple search result images on the screen of the touch panel, so as to highlight the at least one search result image of interest.
On the other hand, Amacker, which relates to evolutionary content determination and management (title) does disclose wherein the display control unit performs control, in response to a user's manipulation action on at least one search result image of interest from among the multiple search result images on the screen of the touch panel, so as to highlight the at least one search result image of interest. (“A user may view detailed information about any item within a search result set displayed in the page. FIG. 5(a) illustrates an example display 500 wherein selection of an item 554 in search result set by a user causes display of an information window 508. The information window 508 is shown in FIG. 5(b) as a pop-up (or modal) window or panel that provides detailed information about the selected item and a "buy it!" box 510. The particular position of window 508 is exemplary. The window 508 may be displayed in any portion of display 550. Any appropriate information about the selected item can be displayed in window 508, such as a user rating, price, product information (e.g., images, video, etc.), links to related information (e.g., third party reviews, etc.), and the like.”)(e.g., figures 5(a) and 5(b) and col 8 lines 11-24).
It would have been obvious to combine Amacker with Lee and Russo to one of ordinary skill in the art before the effective filing date of the claimed invention for the same reasons as set forth in claim 12, above.
Regarding claim 16, Russo in view of Lee and in further view of Amacker discloses the information communication terminal device according to claim 12. Russo further discloses wherein the display control unit performs control, in response to the second manipulation action, such that the display in animated form is achieved in which the speed with which each of the multiple search result images appears at a predetermined rate within a predetermined period of time is increased. (“The sequential display of search results may occur fully automatically, such as by displaying the next search result automatically after a set amount of time (e.g., a new search result is displayed every five seconds), which may be set by the user and/or by software.” User sets speed which can be increased or decreased)(e.g., paragraph [0050]).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Russo in view of Lee and in further view of Hanechak (U.S. Publication No. 2006/0218522 A1, hereinafter referred to as “Hanechak”).
Regarding claim 13, Russo in view of Lee discloses the information communication terminal device according to claim 1. However, neither reference appears to specifically disclose wherein the search query acquisition unit acquires, in response to a user's manipulation action on at least one search result image of interest from among the multiple search result images on the screen of the touch panel, a new search query based on metadata associated with the at least one search result image of interest.
On the other hand, Hanechak, which relates to selecting images using associated keywords (title), does disclose wherein the search query acquisition unit acquires, in response to a user's manipulation action on at least one search result image of interest from among the multiple search result images on the screen of the touch panel, a new search query based on metadata associated with the at least one search result image of interest. (“ER 220 displays image 202 at a larger size, allowing closer inspection by the user, and displays a Show More Like This link 222. ER 220 would also typically include other links allowing the user to perform other actions, as mentioned above in connection with SR 200. If desired by the operator of server 130, ER 220 could also include various additional details about the selected image, such as image copyright information, usage pricing, and a list of the keywords associated with the selected image.”)(e.g., figures 2A-2B and paragraphs [0022]-[0023]).
It would have been obvious to combine Lee with Russo for the same reasons as provided in claim 1, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, Russo does not appear to specifically disclose performing a subsequent search based on keywords associated with the selected query image. On the other hand, Hanechak provides the ability for users to select the image result, and based on the selection, causing additional searches to be performed based on keywords that are associated with the image. E.g., figures 2A-2B and paragraphs [0022]-[0023]. This provides “a method of identifying images of potential interest that lends itself to embodiments allowing use by busy or unsophisticated users.” Paragraph [0008]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate show more like this image feature as disclosed in Hanechak to the Russo-Lee combination to provide the user the ability to better find relevant images of interest.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Russo in view of :ee in further view of Amacker and in further view of Hanechak.
Regarding claim 15, Russo in view of Lee and in further view of Amacker discloses the information communication terminal device according to claim 14. However, none of the references appears to specifically disclose herein the search query acquisition unit acquires a new search query based on metadata associated with the at least one search result image of interest, if the user chooses to perform a search.
On the other hand, Hanechak further discloses wherein the search query acquisition unit acquires a new search query based on metadata associated with the at least one search result image of interest, if the user chooses to perform a search. (“At step 310, thumbnails of the images having the highest ISS scores are retrieved from images 134 and displayed to the user on a review screen SR 200 with associated Show More Like This links to allow the user to initiate another similarity search for one of the newly displayed images, if desired. The number of thumbnails of similar images provided to the user for review is an implementation decision of the operator of server 130.”)(e.g., figure 3 and paragraph [0029]).
It would have been obvious to combine Amacker with Russo and Lee for the same reasons as provided in claim 12, above. Russo discloses searching and indexing of photos based on ink annotations. In Russo, users can search using a touchscreen and search results are shown as a slideshow. However, neither Russo nor Amacker appears to specifically disclose performing a subsequent search based on keywords associated with the selected query image. On the other hand, Hanechak provides the ability for users to select the image result, and based on the selection, causing additional searches to be performed based on keywords that are associated with the image. E.g., figures 2A-2B and paragraphs [0022]-[0023]. This provides “a method of identifying images of potential interest that lends itself to embodiments allowing use by busy or unsophisticated users.” Paragraph [0008]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate show more like this image feature as disclosed in Hanechak to the Russo-Lee-Amacker combination to provide the user the ability to better find relevant images of interest.
Conclusion
The prior art made of record, listed on form PTO-892, and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICHARD L BOWEN whose telephone number is (571)270-5982. The examiner can normally be reached Monday through Friday 7:30AM - 4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aleksandr Kerzhner can be reached at (571)270-1760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICHARD L BOWEN/Primary Examiner, Art Unit 2165