NON-FINAL OFFICE ACTION
REISSUE OF U.S. PATENT NO. 9,565,149
TABLE OF CONTENTS
1. ACKNOWLEDGEMENTS 3
2. ADDITIONAL TERMS 4
3. REISSUE PROCEDURAL REMINDERS 5
4. STATUS OF CLAIMS 6
5. PRIORITY AND AIA STATUS 6
6. INFORMATION CONSIDERED 8
7. PRIOR ART CITED 8
8. BROADEST REASONABLE INTERPRETATION (BRI) 9
9. CLAIM OBJECTIONS 10
10. CLAIM REJECTIONS – 35 USC § 251 (New Matter) 10
11. CLAIM REJECTIONS – 35 USC § 112(a) 10
11.1. New Matter in Claims 14–24 11
11.2. New Matter in Claims 22–24 12
12. CLAIM REJECTIONS – 35 USC § 112(b) 13
12.1. Unclear Reference To “the message” 14
12.2. Claim 14 Includes Contradictory Language 14
12.3. Claim 22 Includes Contradictory Language 15
12.4. Conclusion of 35 USC § 112(b) Rejection 15
13. CLAIM REJECTIONS – 35 USC § 103 16
13.1. Summary of Prior Art Rejections 16
13.2. Ground 1: Obvious Over Gura, in view of Virtanen or Winkler 18
13.3. Ground 2: Obvious Over Gura, in view of Virtanen or Winkler, and in further view of Jensen 26
13.4. Ground 3: Obvious Over Gura, in view of Virtanen or Winkler, and in further view of Fifield 26
13.5. Ground 4: Obvious Over Gura, in view of Virtanen or Winkler, and in further view of Le Bodic 26
13.6. Ground 5: Obvious Over Gura, in view of Virtanen or Winkler, and Jensen, and in further view of Le Bodic 27
13.7. Ground 6: Obvious Over Gura, in view of Virtanen or Winkler, and Fifield, and in further view of Le Bodic 27
14. CONCLUSION 27
15. APPENDIX 29
ACKNOWLEDGEMENTS
This non-final Office action addresses U.S. reissue application No. 18/959,318 (“Instant Application”). Based upon a review of the Instant Application, the actual filing date is 25 November 2024 (“Actual Filing Date”).
The Instant Application is a reissue application of U.S. Patent No. 9,565,149 (“Patent Under Reissue” or “’149 Patent”) titled “MEDIA MESSAGING METHODS, SYSTEMS, AND DEVICES.”
An application for the Patent Under Reissue was filed on 05 November 2013 (“Base Application Filing Date”) and assigned by the Office non-provisional U.S. patent application number 14/072,418 (“Base Application” or “’418 Application”) and issued on 07 February 2017 with claims 1–13 (“Originally Patented Claims”).
The ’149 Patent was subject to an inter partes review (“IPR”). See IPR2022-00210 (“’210 IPR”). The results of the ’210 IPR are published in IPR certificate no. US 9,565,149 K1, which is now attached to the ’210 Patent. The results show that the Originally Patent Claims are cancelled. Id.
On 22 July 2025, and because all claims of the original patent are cancelled, Applicant filed the necessary Petition Under 37 C.F.R. § 1.182 To Proceed With Reissue of a Patent Having All Claims Cancelled.
On 29 December 2025, the Office mailed a Decision Granting Petition, which states the “petition is granted and prosecution of the ‘318 reissue application will proceed.” See page 6 (emphasis in original).
On 02 July 2026, Applicant filed a petition entitled “PETITION UNDER 37 C.F.R. § 1.182 TO ADVANCE APPLICATION OUT-OF-TURN DUE TO IRREPARABLE HARM TO APPLICANT CAUSED BY EXCESSIVE DELAY.” This petition is acknowledged.
ADDITIONAL TERMS
The following terms may appear in this Office action and, unless expressly noted otherwise, are defined as follows:
“POSITA” refers to a Person of Ordinary Skill in the Art.
“BRI” refers to Broadest Reasonable Interpretation.
“MPEP” refers to Manual of Patent Examining Procedure, Ninth Edition, Rev. 01.2024.
“IFW” refers to Image File Wrapper.
“35 USC” refers to Title 35 of the United States Code.
“37 CFR” refers to Title 37 of the Code of Federal Regulations.
“AIA ” refers to America Invents Act.
“Original Application” means the prosecution history of the Base Application, including the applications in the patent family’s entire prosecution history. See MPEP § 1412.02.
“Original Disclosure” means the substantive sections of the Base Application (i.e., the abstract, drawings, specification, and original claims) that were present in the Base Application on the Base Application Filing Date.
“Applicant” (uppercase) refers to the Applicant of the Instant Application.
“applicant” (lowercase) refers to an applicant(s) generally.
“patent owner” (lowercase) refers to a patent owner(s) generally and not the Applicant.
“Examiner” (uppercase) refers to the Examiner of the Instant Application.
“examiner” (lowercase) refers to an examiner(s) generally, e.g. the examiner of the Base Application, or any examiner(s) other than the Examiner.
REISSUE PROCEDURAL REMINDERS
Disclosure of other proceedings. Applicant is reminded of the continuing obligation under 37 CFR § 1.178(b), to timely apprise the Office of any prior or concurrent proceed-ing in which the Patent Under Reissue is or was involved. These proceedings would include interferences, reissues, reexaminations, and litigation.
Disclosure of material information. Applicant is further reminded of the continuing obligation under 37 CFR § 1.56, to timely apprise the Office of any information which is material to patentability of the claims under consideration in this reissue appli-cation.
These disclosure obligations rest with each individual associated with the filing and prosecution of this application for reissue. See also MPEP §§ 1404, 1442.01 and 1442.04.
Manner of making amendments. Applicant is reminded that changes to the Instant Application must comply with 37 CFR § 1.173, such that all amendments are made in respect to the Patent Under Reissue as opposed to any prior changes entered in the Instant Application. All added material must be underlined, and all omitted material must be enclosed in brackets, in accordance with Rule 173. Applicant may submit an appendix to any response in which claims are marked up to show changes with respect to a previous set of claims, however, such claims should be clearly denoted as “not for entry.”
STATUS OF CLAIMS
Claims 14–24, as set forth in the preliminary amendment filed 22 July 2025, are currently pending (“Pending Claims”) and currently examined (“Examined Claims”).
Regarding the Examined Claims and as a result of this Office action:
Claims 14–24 are rejected under 35 USC § 112(a).
Claims 14–24 are rejected under 35 USC § 112(b).
Claims 14–24 are rejected under 35 USC § 251.
Claims 14–24 are rejected under 35 USC § 103.
PRIORITY AND AIA STATUS
Domestic Priority. Based upon a review of the Instant Application and the Patent Under Reissue, the Examiner finds that in the Instant Application there is a claim for benefit of domestic priority under 35 USC §§ 120 or 119(e) to provisional application no. 61/722,560, filed on 05 November 2012 (“’560 Provisional Application”), and to provisional application no. 61/835,975, filed on 17 June 2013 (“’975 Provisional Application”). To the extent the disclosure of the ’560 Provisional Application supports the Pending Claims under 35 USC § 112, the supported claims receive benefit of the filing date of the ’560 Provisional Application. Likewise, to the extent the disclosure of the ’975 Provisional Application supports the Pending Claims under 35 USC § 112, the supported claims receive benefit of the filing date of the ’975 Provisional Application.
Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date as follows:
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 USC § 112(a) or the first paragraph of pre-AIA 35 USC § 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
The disclosure of the ’560 Provisional Application and the ’975 Provisional Application fails to provide adequate support or enablement in the manner provided by 35 USC § 112(a) for one or more claims of the Pending Claims. Particularly, the ’560 Provisional Application and the ’975 Provisional Application fails to provide adequate support in the manner provided by 35 USC § 112(a) for the same reasons as presented in section CLAIM REJECTIONS – 35 USC § 112(a), below.
Accordingly, the Examiner finds the effective filing date of the Pending Claims to be the Actual Filing Date.
AIA Status. Because the Instant Application does not contain a claim having an effective date before March 16, 2013, the AIA provisions apply. In the event the determination of the status of the application as subject to AIA 35 USC §§ 102 and 103 is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
INFORMATION CONSIDERED
The information disclosure statement (“IDS”) filed on the Actual Filing Date is in compliance with the provisions of 37 CFR § 1.97 and § 1.98. Accordingly, the IDS has been considered by the Examiner as noted on the attached annotated document.
In accordance with MPEP § 1406, the Examiner has reviewed and considered the prior art cited or “of record” in the original prosecution of the Patent Under Reissue. Applicant is reminded that a listing of the information cited or “of record” in the original prosecution of the Patent Under Reissue has been considered in the Instant Application and need not be resubmitted.
PRIOR ART CITED
The following prior art patents and printed publications are cited below1:
International Publication Number WO 2007/080570 (“Gura” or “EX1004”);
C. Jensen, et al., “Multidimensional Databases and Data Warehousing” (2010), excerpts (“Jensen” or “EX1005”);
U.S. Patent Application Publication 2007/0033212 (“Fifield” or “EX1006”);
G. Le Bodic, “Mobile Messaging Technologies and Services: SMS, EMS and MMS” (2d ed. 2005), excerpts (“Le Bodic” or “EX1007”);
U.S. Patent Application Publication 2014/0115070 (“Virtanen”); and
U.S. Patent Application Publication 2002/0137507 (“Winkler”).
The prior art made of record and considered pertinent to Applicant’s disclosure, but not relied upon to reject the claims in this Office action, is listed on the attached document titled “Notice of References Cited” (PTO-892). Unless expressly noted otherwise by the Examiner, all documents listed on the PTO-892 are cited in their entirety.
BROADEST REASONABLE INTERPRETATION (BRI)
During examination, claims are given the broadest reasonable interpretation consistent with the specification and limitations in the specification are not read into the claims. See MPEP § 2111, MPEP § 2111.01 and In re Yamamoto et al., 222 USPQ 934 (Fed. Cir. 1984). Under a broadest reasonable interpretation, words of the claim must be given their plain meaning, unless such meaning is inconsistent with the specification. See MPEP § 2111.01 I. Moreover, it is improper to import claim limitations from the specification, e.g., a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment. See MPEP § 2111.01 II. Therefore, unless otherwise noted below, the Examiner will interpret the limitations of the Pending Claims using the broadest reasonable interpretation.
After careful review of the original specification, the Examiner finds he cannot locate any lexicographic definitions (either express lexicographic definitions or implied lexicographic definitions) with the required clarity, deliberateness, and precision. Because the Examiner cannot locate any lexicographic definitions with the required clarity, deliberateness, and precision, the Examiner concludes that Applicant is not his own lexicographer. See MPEP § 2111.01 IV.
CLAIM OBJECTIONS
Claim 22 is objected to for reciting “the first user interface” (in the “inserting” clause) instead of “a first user interface.” Appropriate correction is required.
Claim 22 is also objected to for reciting “the rich media data request” (in the last line) instead of “the at least one media insertion request.” Appropriate correction is required.
CLAIM REJECTIONS – 35 USC § 251 (New Matter)
Claims 14–24 are rejected under 35 USC § 251 as being based upon new matter added to the patent for which reissue is sought. The added material which is not supported by the prior patent is described below in § CLAIM REJECTIONS – 35 USC § 112(a).
CLAIM REJECTIONS – 35 USC § 112(a)
The following is a quotation of 35 USC § 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 14–24 are rejected under 35 USC § 112(a) as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, at the time the application was filed, had possession of the claimed invention.
New Matter in Claims 14–24
Claim 14 does not comply with the written description requirement because it contains new matter. Particularly, for the reasons set forth below, the following limitation(s) introduces new matter,
a messaging service configured to:
receive a message, the message including a rich media data request from the first user device;
retrieve the requested rich media data from the at least one database of rich media data;
insert the retrieved rich media data into the message on the first user interface while maintaining text flow, wherein the retrieved rich media data is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message;
The ’149 Patent discloses,
Rather than the user attaching an image of a “pie chart graph” to the bottom of the email (which must then be opened by receiver separately taking reader away from text), the user (sender) can simply type “!pc” within the flow of their email typing and the image of said “pie chart graph” will automatically “place itself within the paragraph of the words being typed, the symbol “!pc” having been pre-assigned to said image of “pie chart graphic” within user's device or photograph library.
See ’149 Patent at C2:L18–27 (emphasis added); and
“The assigned symbols, letters or numbers can be typed while texting, resulting in immediate and automatic placement of the associated Photocon where the symbol, number or letter was typed.” See ’149 Patent at C4:L64–67 (emphasis added).
However, the Examiner finds that the above noted portions of the ’149 Patent alone, in combination with each other, and/or in combination with any other disclosure of the ’149 Patent, do not provide adequate written description for the now claimed “messaging service,” supra. In particular, the above portions disclose that a media item “will automatically place itself within the paragraph of the words being typed” or be placed “where the symbol was typed” in a manner that is “immediate and automatic.” In this disclosure, there is no express or inherent involvement of a “messaging service configured to: receive a message, […]; retrieve the requested rich media data […]; and insert the retrieved rich media data into the message […] while the sending user is typing the message,” as now recited by new claim 14. Therefore, the “messaging service” as now claimed in new claim 14 is new matter.
Dependent claims 15–21 fail to cure this deficiency of independent claim 14 (set forth directly above) and are rejected accordingly.
Claims 22–24 contain language similar to claims 14–21 as discussed in the preceding paragraphs, and for reasons similar to those discussed above, claims 22–24 are also rejected under 35 USC § 112 as failing to comply with the written description requirement. For example, claim 22 requires a method comprising “inserting the retrieved media into the message on the electronic device of the sending user […] while the sending user is typing the message” in combination with “evaluating the at least one media insertion request at a messaging cloud, the messaging cloud including a Media Messaging Platform; [and] retrieving media from at least one Media Messaging Platform database of media based on the evaluated at least one media insertion request.” However, similar to the reasoning above for claim 14, the ’149 Patent does not disclose a method where the media is inserted while the user is typing and where a media insertion request is evaluated “at a messaging cloud,” as now recited by new claim 22. Thus, the combination of these limitations is new matter.
New Matter in Claims 22–24
Claim 22 does not comply with the written description requirement because it contains new matter. Particularly, for the reasons set forth below, the following limitation(s) introduces new matter, “receiving a message on the electronic device of the sending user, the message including at least one media insertion request […] and the message session formatted on the electronic device of the sending user such that the rich media data request is within the message.”
The ’149 Patent discloses that the messaging service receives a message including a media insertion request. See, e.g., Fig. 13 showing arrow (transmission) of “MMP Messages or Texts” from a sending user device 320 to a server 202. See also C9:L23–35, “In a third step, the first user types, edits, or otherwise composes a message with MMP tags within the message. That message is transmitted to MMP Communication System 202, which uses, for example, the picture or image in the photocon library, to insert into the message, which is then transferred to the second user 103 via Design Structure 320.” Thus, it is disclosed that that a sending user device is where a message with a media insertion request is composed, but not where a message with a media insertion request is “receiv[ed],” as now claimed. It is the server that receives such a message. Therefore, the noted limitation is new matter.
Dependent claims 23–24 fail to cure this deficiency of independent claim 22 (set forth directly above) and are rejected accordingly.
CLAIM REJECTIONS – 35 USC § 112(b)
The following is a quotation of 35 USC § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 14–24 are rejected under 35 USC § 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Unclear Reference To “the message”
New claim 22 recites “presenting the message to the receiving user device,” yet previously recites “receiving a message […] the message including at least one media insertion request” and “inserting the retrieved media into the message.” Claim 22 is indefinite because it is unclear whether “the message” presented in the message with the at least one media insertion request or the message with the retrieved media or both.
Dependent claims 23–24 fail to cure this deficiency of independent claim 22 (set forth directly above) and are rejected accordingly.
Claims 14–21 contain language similar to claims 22–24 as discussed in the preceding paragraphs, and for reasons similar to those discussed above, claims 14–21 are also rejected under 35 USC § 112 as being indefinite.
Claim 14 Includes Contradictory Language
Claim 14 is indefinite because it contains contradictory limitations. First, claim 14 includes limitations that require a message to be received at a messaging service (“messaging service configured to: receive a message, the message including a rich media data request from the first user device”). Because the message is received, the message can be interpreted as complete. However, claim 14 also includes limitations that require rich media data to be inserted into the message “on the first user interface while maintaining text flow […] while the sending user is typing the message,” which contradicts the interpretation that the received message is complete. Thus, claim 14 is indefinite.
In applying the prior art, the Examiner will interpret the above limitations as follows.
Interpretation (1): only a portion of the message is received at the service and the rich media data is inserted into the message by the service by sending the rich media data to the sending user device for insertion, or
Interpretation (2): the entire message is received and the rich media data is inserted into the message by the service and then the service returns the message (with the rich media data) to the sending user device for further composition.
Dependent claims 15–21 fail to cure this deficiency of independent claim 14 (set forth directly above) and are rejected accordingly.
Claim 22 Includes Contradictory Language
Claim 22 is indefinite because it contains contradictory limitations. First, claim 22 includes a preamble that limits the claimed method to a method performed on an electronic device of a sending user (“[a] method of streamlined media insertion on an electronic device of a sending user, the method comprising.”). However, claim 22 also includes limitations that require certain steps to be performed by a cloud device other than the electronic device of a sending user, e.g., “evaluating the at least one media insertion request at a messaging cloud,” which contradicts the interpretation that the method is performed “on an electronic device of a sending user.” Thus, claim 22 is indefinite.
Dependent claims 23–24 fail to cure this deficiency of independent claim 22 (set forth directly above) and are rejected accordingly.
Conclusion of 35 USC § 112(b) Rejection
The Examiner has considered all limitations in the Examined Claims even though some claim limitations are indefinite. See MPEP § 2143.03 I. ¶1 noting that “[a] claim limitation which is considered indefinite cannot be disregarded.” However, the Examiner concludes that because claims 14–24 are indefinite under § 112(b), these claims, by definition, cannot be properly construed. See e.g. Honeywell International Inc. v. ITC, 341 F.3d 1332, 1342 (Fed. Cir. 2003) (“Because the claims are indefinite, the claims, by definition, cannot be construed.”). Therefore, in accordance with MPEP § 2173.06 and the USPTO’s policy of trying to advance prosecution by providing prior art rejections (or indicating allowance of the claims) even though certain claims are indefinite, these indefinite claims are construed and the prior art is currently applied as much as practically possible. Applicant is reminded that when a particular § 112(b) rejection is overcome by Applicant, and the claim limitation can then be properly construed, the Examiner will reevaluate the prior art using this proper claim construction and then reassess the patentability of the claim over the prior art.
CLAIM REJECTIONS – 35 USC § 103
The following is a quotation of 35 USC § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Summary of Prior Art Rejections
This is not a broadening reissue, so new independent claims 14 and 22 include all of the limitations of original patent claims 1 and 10, respectively, plus additional narrowing limitations.
As stated by Applicant,
In particular, new independent claim 14 substantially repeats the text of cancelled claim 1 and adds the limitation “wherein the retrieved rich media data is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message.” This limitation was not contained in any original claim, and therefore new claim 14 is narrower than all claims in the original patent. New dependent claims 15-20, which substantially repeat the text of cancelled claims that depended from cancelled claim 1, all contain this limitation by virtue of their dependency from claim 14, and are therefore narrower than all claims in the original patent as well. MPEP 1449.01(I)(A)(B)(5)(a).
Similarly, new independent claim 22 substantially repeats the text of cancelled claim 10 and adds the limitation “wherein the retrieved media is inserted and displayed to the sending user on the first user interface.in the message while the sending user is typing the message.” This limitation was not contained in any original claim, and therefore new claim 22 is narrower than all claims in the original patent. New dependent claims 23 and 24, which substantially repeat the text of cancelled claims that depended from cancelled claim 10, contain this limitation by virtue of their dependency from claim 22, and are therefore narrower than all claims in the original patent as well. MPEP 1449.01(I)(A)(B)(5)(a).
See Remarks filed 22 July 2025 (page 8).
In view of the fact that new independent claims 14 and 22 “substantially repeat[]” original patent claims 1 and 10, respectively, the prior art rejections in this Office Action generally follow the prior art rejections discussed in the ’210 IPR. The table below is a summary of the six grounds from the ’210 IPR.
PNG
media_image1.png
380
806
media_image1.png
Greyscale
Moreover, the APPENDIX (at the end of this Office action) includes a copy of the six grounds of rejection as presented by the Petitioner in the ’210 IPR.
In the subsections that follow, the Examiner presents substantially the same six grounds of rejection in rejecting new claims 14–27 with additional teachings of Virtanen and Winkler to address the narrowing limitations added in the Instant Application.
Ground 1: Obvious Over Gura, in view of Virtanen or Winkler
Claims 14–24 are rejected under 35 USC § 103 as being unpatentable over Gura, in view of Virtanen or Winkler.
As per claim 14, Gura teaches the limitations of original patent claim 1 now present in new claim 14 as shown in the APPENDIX (see §§ VII.A and VII.B).
Gura does not expressly teach “insert the retrieved rich media data into the message on the first user interface while maintaining text flow, wherein the retrieved rich media data is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message,” as now claimed.
As noted above in the subsection entitled “Claims Include Contradictory Language,” the limitation above is subject to Interpretation (1) and Interpretation (2). Both are addressed below with prior art.
Interpretation (1)
Virtanen teaches “the user is able to compose a message and, part way through the message, insert a content item in the message without breaking the flow of typing […].” See ¶ [0118]. To achieve this, Virtanen teaches a “means for using, during message composition using a keyboard, one or more of the keyboard entered characters forming part of the message to search for one or more content items having metadata matching the one or more entered characters; and means for providing the one or more content items having matching metadata for insertion or attachment to the message.” See ¶¶ [0050]–[0051]. Virtanen further teaches “an apparatus/device is able to access stored content items which are available for inclusion (attachment to, or insertion in) a message. Such content items may be stored on a memory of the apparatus/device, or may be stored remotely, for example on a server or cloud which the apparatus/device can access. At least one of these content items has at least one metadata label/tag associated with it.” See ¶ [0084]. Thus, in the case of remote storage, e.g., at a sever or cloud, Virtanen teaches that the server or cloud would use part of the message to search for rich media data having metadata matching entered characters and provide the discovered rich media data for insertion into the message.
In view of the above, Virtanen teaches “insert the retrieved rich media data into the message on the first user interface while maintaining text flow, wherein the retrieved rich media data is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message,” as now claimed and interpreted under Interpretation (1).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains, before the effective filing date of the claimed invention, to modify Gura’s remote system (e.g., see discussion of Figs. 1–2 in APPENDIX) to include inserting the retrieved rich media data into the message, as taught by Virtanen, i.e., to receive part of the message as the user is typing and search for the relevant rich media data and provide the matching rich media data for insertion into the message, as taught by Virtanen. One would have been motivated to do so because the user in Virtanen’s system is given more flexibility during message composition. For example, Virtanen teaches that user selectable rich media data choices matching the message portion can be provided for a selection by the user (e.g., see ¶ [0019]). Moreover, one would have been motivated to do so because the user in Virtanen’s system can see how the message will look with the inserted rich media data prior to sending the message to the recipient.
Interpretation (2)
Winkler teaches “According to the present invention it is suggested to provide the users with new multimedia service application 16 (‘graphical engine’) located at a designated server associated with the user gateway server and/or SMS center.” See ¶ [0016]. Winkler further teaches “The text message is processed and analyzed for identifying the keywords in the text, the text style or manner (ie. slang sarcasm, happiness) and the message text context and subject. […]. Once the message is categorized according to one of the available subjects in the service database, the application selects the respective multimedia items such as graphical image or animation in relation to the message subject context or identified keywords.” See ¶ [0019]. Winkler further teaches “The selected multimedia items are merged in the original message, creating modified message that combines the text and multimedia according to identified key words and text context. […], the modified message is transferred back to the originator user for preview and confirmation. The originator user views the modified message and may edit the message text or choose from selection of multimedia items provided by the graphics engine.” See ¶ [0020].
In view of the above, Winkler teaches “insert the retrieved rich media data into the message on the first user interface while maintaining text flow, wherein the retrieved rich media data is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message,” as now claimed and interpreted under Interpretation (2).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains, before the effective filing date of the claimed invention, to modify Gura’s remote system (e.g., see discussion of Figs. 1–2 in APPENDIX) to include inserting the retrieved rich media data into the message, as taught by Winkler, i.e., to receive the entire message, insert the rich media data, and return the message with rich media data to the user for further composition, as taught by Winkler. One would have been motivated to do so because the user in Winkler’s system is given more flexibility during message composition. For example, Winkler teaches that user selectable rich media data choices matching the message can be provided for a selection by the user (e.g., see ¶ [0020]). Moreover, one would have been motivated to do so because the user in Winkler’s system can see how the message will look with the inserted rich media data prior to sending the message to the recipient.
As per claim 15, Gura and Virtanen or Winkler teaches the limitations of original patent claim 2 now present in new claim 15 as shown in the APPENDIX (see Gura as cited in § VII.C).
As per claim 16, Gura and Virtanen or Winkler teaches the limitations of original patent claim 4 now present in new claim 16 as shown in the APPENDIX (see Gura as cited in § VII.E).
As per claim 17, Gura and Virtanen or Winkler teaches the limitations of original patent claim 5 now present in new claim 17 as shown in the APPENDIX (see Gura as cited in § VII.F).
As per claim 18, Gura and Virtanen or Winkler teaches the limitations of original patent claim 6 now present in new claim 18 as shown in the APPENDIX (see Gura as cited in § VII.G).
As per claim 19, Gura and Virtanen or Winkler teaches the limitations of original patent claim 7 now present in new claim 19 as shown in the APPENDIX (see Gura as cited in § VII.H).
As per claim 20, Gura and Virtanen or Winkler teaches the limitations of original patent claim 8 now present in new claim 20 as shown in the APPENDIX (see Gura as cited in § VII.I).
As per claim 21, Gura and Virtanen or Winkler teaches the limitations of original patent claim 9 now present in new claim 21 as shown in the APPENDIX (see Gura as cited in § VII.J).
As per claim 22, Gura teaches the limitations of original patent claim 10 now present in new claim 22 as shown in the APPENDIX (see § VII.K).
Gura does not expressly teach “inserting the retrieved media into the message on the electronic device of the sending user while maintaining text flow, wherein the retrieved media is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message,” as now claimed.
Although claim 22 is merely broad and not indefinite for the same reasons as claim 14, as noted above in the subsection entitled “Claims Include Contradictory Language,” claim 22 does not specify whether the “at least one media insertion request” is sent to the “messaging cloud” with the entire message or by itself, and thus the limitation above is also subject to Interpretation (1) and Interpretation (2). Both are addressed below with prior art.
Interpretation (1)
Virtanen teaches “the user is able to compose a message and, part way through the message, insert a content item in the message without breaking the flow of typing […].” See ¶ [0118]. To achieve this, Virtanen teaches a “means for using, during message composition using a keyboard, one or more of the keyboard entered characters forming part of the message to search for one or more content items having metadata matching the one or more entered characters; and means for providing the one or more content items having matching metadata for insertion or attachment to the message.” See ¶¶ [0050]–[0051]. Virtanen further teaches “an apparatus/device is able to access stored content items which are available for inclusion (attachment to, or insertion in) a message. Such content items may be stored on a memory of the apparatus/device, or may be stored remotely, for example on a server or cloud which the apparatus/device can access. At least one of these content items has at least one metadata label/tag associated with it.” See ¶ [0084]. Thus, in the case of remote storage, e.g., at a sever or cloud, Virtanen teaches that the server or cloud would use part of the message to search for rich media data having metadata matching entered characters and provide the discovered rich media data for insertion into the message.
In view of the above, Virtanen teaches “inserting the retrieved media into the message on the electronic device of the sending user while maintaining text flow, wherein the retrieved media is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message,” as now claimed and interpreted under Interpretation (1).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains, before the effective filing date of the claimed invention, to modify Gura’s remote system (e.g., see discussion of Figs. 1–2 in APPENDIX) to include inserting the retrieved rich media data into the message, as taught by Virtanen, i.e., to receive part of the message as the user is typing and search for the relevant rich media data and provide the matching rich media data for insertion into the message, as taught by Virtanen. One would have been motivated to do so because the user in Virtanen’s system is given more flexibility during message composition. For example, Virtanen teaches that user selectable rich media data choices matching the message portion can be provided for a selection by the user (e.g., see ¶ [0019]). Moreover, one would have been motivated to do so because the user in Virtanen’s system can see how the message will look with the inserted rich media data prior to sending the message to the recipient.
Interpretation (2)
Winkler teaches “According to the present invention it is suggested to provide the users with new multimedia service application 16 (‘graphical engine’) located at a designated server associated with the user gateway server and/or SMS center.” See ¶ [0016]. Winkler further teaches “The text message is processed and analyzed for identifying the keywords in the text, the text style or manner (ie. slang sarcasm, happiness) and the message text context and subject. […]. Once the message is categorized according to one of the available subjects in the service database, the application selects the respective multimedia items such as graphical image or animation in relation to the message subject context or identified keywords.” See ¶ [0019]. Winkler further teaches “The selected multimedia items are merged in the original message, creating modified message that combines the text and multimedia according to identified key words and text context. […], the modified message is transferred back to the originator user for preview and confirmation. The originator user views the modified message and may edit the message text or choose from selection of multimedia items provided by the graphics engine.” See ¶ [0020].
In view of the above, Winkler teaches “inserting the retrieved media into the message on the electronic device of the sending user while maintaining text flow, wherein the retrieved media is inserted and displayed to the sending user on the first user interface in the message while the sending user is typing the message,” as now claimed and interpreted under Interpretation (2).
Therefore, it would have been obvious to a person having ordinary skill in the art to which the claimed invention pertains, before the effective filing date of the claimed invention, to modify Gura’s remote system (e.g., see discussion of Figs. 1–2 in APPENDIX) to include inserting the retrieved rich media data into the message, as taught by Winkler, i.e., to receive the entire message, insert the rich media data, and return the message with rich media data to the user for further composition, as taught by Winkler. One would have been motivated to do so because the user in Winkler’s system is given more flexibility during message composition. For example, Winkler teaches that user selectable rich media data choices matching the message can be provided for a selection by the user (e.g., see ¶ [0020]). Moreover, one would have been motivated to do so because the user in Winkler’s system can see how the message will look with the inserted rich media data prior to sending the message to the recipient.
As per claim 23, Gura and Virtanen or Winkler teaches the limitations of original patent claim 11 now present in new claim 23 as shown in the APPENDIX (see Gura as cited in § VII.L).
As per claim 24, Gura and Virtanen or Winkler teaches the limitations of original patent claim 13 now present in new claim 24 as shown in the APPENDIX (see Gura as cited in § VII.N).
Ground 2: Obvious Over Gura, in view of Virtanen or Winkler, and in further view of Jensen
Claims 14–24 are rejected under 35 USC § 103 as being unpatentable over Gura, in view of Virtanen or Winkler, and in further of view of Jensen.
As per claims 14–24, these claims are alternatively met by Gura, in view of Virtanen or Winkler, and in further of view of Jensen, for the same reasons that original patent claims 1–13 are alternatively rejected using Jensen in the below APPENDIX (see Jensen as cited in § VIII.A).
Ground 3: Obvious Over Gura, in view of Virtanen or Winkler, and in further view of Fifield
Claims 14–24 are rejected under 35 USC § 103 as being unpatentable over Gura, in view of Virtanen or Winkler, and in further view of Fifield.
As per claims 14–24, these claims are alternatively met by Gura, in view of Virtanen or Winkler, and in further of view of Fifield, for the same reasons that original patent claims 1–13 are alternatively rejected using Fifield in the below APPENDIX (see Fifield as cited in § VIII.B).
Ground 4: Obvious Over Gura, in view of Virtanen or Winkler, and in further view of Le Bodic
Claim 19 is rejected under 35 USC § 103 as being unpatentable over Gura, in view of Virtanen or Winkler, and in further view of Le Bodic.
As per claim 19, this claim is alternatively met by Gura, in view of Virtanen or Winkler, and in further of view of Le Bodic, for the same reasons that original patent claim 7 is alternatively rejected using Le Bodic in the below APPENDIX (see Le Bodic as cited in § IX).
Ground 5: Obvious Over Gura, in view of Virtanen or Winkler, and Jensen, and in further view of Le Bodic
Claim 19 is rejected under 35 USC § 103 as being unpatentable over Gura, in view of Virtanen or Winkler, and Jensen, and in further view of Le Bodic.
As per claim 19, this claim is alternatively met by Gura, in view of Virtanen or Winkler, and Jensen, and in further of view of Le Bodic, for the same reasons that original patent claim 7 is alternatively rejected using Le Bodic in the below APPENDIX (see Le Bodic as cited in § IX).
Ground 6: Obvious Over Gura, in view of Virtanen or Winkler, and Fifield, and in further view of Le Bodic
Claim 19 is rejected under 35 USC § 103 as being unpatentable over Gura, in view of Virtanen or Winkler, and Fifield, and in further view of Le Bodic.
As per claim 19, this claim is alternatively met by Gura, in view of Virtanen or Winkler, and Fifield, and in further of view of Le Bodic, for the same reasons that original patent claim 7 is alternatively rejected using Le Bodic in the below APPENDIX (see Le Bodic as cited in § IX).
CONCLUSION
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to Jacob Coppola whose telephone number is 571-270-3922.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, Andrew J. Fischer can be reached at 571-272-6779. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of this proceeding may be obtained from the USPTO’s Patent Center. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (in USA or Canada) or 571-272-1000.
General inquiries may also be directed to the Central Reexamination Unit customer service line at (571) 272-7705.
/JACOB C. COPPOLA/Primary Examiner, Art Unit 3992
Conferees:
/KENNETH WHITTINGTON/Primary Examiner, Art Unit 3992
/ANDREW J. FISCHER/Supervisory Patent Examiner, Art Unit 3992
APPENDIX
The below APPENDIX includes a copy of sections VII, VIII, and IX from the Petition filed in the ’210 IPR. The Table below is a summary of the rejections included in the APPENDIX.
PNG
media_image1.png
380
806
media_image1.png
Greyscale
THE CHALLENGED CLAIMS ARE RENDERED OBVIOUS BY GURA (GROUND 1)
Gura discloses an enhanced messaging system that receives text messages and enhances them with various types of media items in the same manner described in the ’149 Patent. Significantly, Gura also discloses identifying images for insertion in a message based on the combination of the user’s “emotion” and the “identity of the recipient,” as claimed in the ’149 Patent. Thus, Gura discloses the functionality that the examiner found missing in the prior art cited during prosecution.
As described in more detail below, Gura discloses and renders obvious to a POSITA every element of the challenged claims.
Overview of Gura
Gura describes an enhanced messaging system “for image insertion into text messaging, such as a cellular telephone SMS….” EX1004, 1:5-7. By 2006, when Gura was filed, “[t]ext message users [were] well known for using emoticons, short combinations of punctuation marks that indicate emotions in a message” such as “the colon, dash, close parenthesis :-) indicating happiness.” Id., 1:15-17. Gura states that “[u]sers of equivalent messaging services over the Internet, IM or Internet messaging, [were] used to these symbols being automatically translated into corresponding images….” Id., 1:17-19. Gura’s enhanced messaging system provides similar automatic translation to images and other types of digital media for text messaging. See, e.g., id., 4:30-5:1, 10:20-22.
Figure 1a provides an example of a text message modified by Gura’s enhanced messaging system. EX1004, 4:1-2, 5:15-16.
PNG
media_image2.png
584
702
media_image2.png
Greyscale
Gura, Fig. 1a
In the example, “text message 10” sent using “first cellular telephone 12” is “enhanced by the addition of images 16 and 18 to form enhanced message 20.” Id., 5:17-19. Thus, as in the ’149 Patent, Gura’s enhanced messaging system automatically converts in-line text strings to corresponding multimedia items. EX1003 ¶118.
Gura’s enhanced messaging system includes “text enhancement apparatus 100,” which receives text messages and “outputs an appropriately formatted enhanced message.” EX1004, 4:3-5, 6:5-6.
PNG
media_image3.png
452
721
media_image3.png
Greyscale
Gura, Fig. 1b
As illustrated in Figure 1b, the text message enhancement apparatus “receives text message inputs from multiple input devices/applications, 110.1-110.5, and outputs the enhanced message formatted for the particular device/application on which the enhanced message will be displayed, such as cellular telephone 120.1 or IM application 120.5.” Id., 6:5-9.
An embodiment of the text message enhancement apparatus is illustrated in more detail in Figure 2. EX1004, 4:6-7, 6:10-12.
PNG
media_image4.png
593
781
media_image4.png
Greyscale
Gura, Fig. 2
Gura explains that the illustrated embodiment is “located in the cellular network and is associated with the servers that provide SMS and MMS features to the network, namely the Short Message Service Center (SMSC) 20 and Multimedia Service Center (MMSC) 22” (i.e., the apparatus communicates with servers in the existing cellular network). Id., 6:12-15; EX1003 ¶121. In the embodiment, “[a] message interpreter 24 receives the SMS message and analyzes it for suitable enhancements.” EX1004, 6:15-16. That analysis of the text message “may involve analyzing the text of the message being sent for keywords or key phrases, based on which images may be selected.” Id., 16-19.
Image database 30 stores the images and other media that is available to enhance the message, and “[a]ccording to the indexing used in the image database 30, the text of the original message is correlated with images in the image database.” EX1004, 7:28-29. Gura provides a “Pictures Administration & Provisioning Interface” used to add or delete pictures and keywords in the database. Id., 11:16-17, 13:3-7, 15:1-2, 15:14-17, 25:22-26, 27:1-3, 28:7-8. “The database is connected to an image inserter 32 which takes the images selected at the database and inserts them into the text messages.” Id., 7:21-22. The resulting message is “an enhanced version of the original text message” that “is then sent on to the recipient.” Id., 7:22-24.
Gura’s enhanced messaging system analyzes text messages for “keywords or key phrases,” where “keywords indicating sentiment may be used to select images illustrating the corresponding emotion.” EX1004, 6:16-19. Gura also discloses analyzing the “context” of the message, including the “identity of the recipient,” to select the appropriate media item. Id., 6:2-22. For example, “images or sets of images may be used for designated close personal contacts.” Id., 6:23-24. Thus, as in the ’149 Patent, Gura describes an enhanced messaging system for personalizing messages by inserting multimedia items based on keywords indicating the user’s emotional state and contact information about the recipient. EX100¶126.
Claim 1
A communications system comprising:
Gura discloses a communications system.2 EX1003 ¶¶135-138. In particular, Gura discloses an enhanced messaging system for “image insertion into text messaging, such as a cellular telephone SMS….” EX1004, 1:5-7; see also id., 4:30-32 (“embodiments comprise an apparatus and a method for enhancement of a text message as it passes over a network, typically a cellular network”), 5:17-25, 6:5-15, 11:1-6, Fig. 2. In addition to enhancing SMS text messages in a cellular network, Gura’s system can also enhance “[a]pplication-to-application messages (i.e., instant messaging)” and “email messages.” Id., 5:23-6:4; see also id., Fig. 1b. Thus, Gura discloses a communications system (e.g., enhanced text messaging, enhanced instant messaging, enhanced email messaging), as recited in the preamble of claim 1.
a first user device presenting a first user interface;
Gura discloses this limitation. EX1003 ¶¶140-143. For instance, Figure 1a illustrates a text message being enhanced by Gura. EX1004, 4:1-2, 5:15-16.
PNG
media_image5.png
580
1048
media_image5.png
Greyscale
Gura, Fig. 1a (annotated)
“A text message 10 is sent by the user of a first cellular telephone 12,” which constitutes a first user device. Id., 5:17; EX1003 ¶142. The first cellular telephone 12 is presenting a first user interface that displays text message 10 (and is shown on the illustrated cellular telephone). Id. ¶¶142-143; see also EX1004, 6:5-9, Fig. 1b.
a second user device presenting a second user interface;
Gura discloses this limitation. EX1003 ¶¶145-148. In Figure 1a, the enhanced message is received by a second cellular telephone, which constitutes a second user device. EX1004, 5:15-22; EX1003 ¶147.
PNG
media_image6.png
580
1020
media_image6.png
Greyscale
Gura, Fig. 1a (annotated)
Gura explains that “the message is enhanced by the addition of images 16 and 18 to form enhanced message 20” received by the second cellular telephone. EX1004, 5:17-19. And like the first cellular telephone 12, the second cellular telephone receiving the message is presenting a second user interface that displays enhanced text message 20 (and is shown on the illustrated cellular telephone). EX1003 ¶¶147-148.
Figure 1b also illustrates a first user device and a second user device. EX1003 ¶¶141-146. There, “text enhancement apparatus 100 receives text message inputs from multiple input devices/application 110.1-110.5 and outputs the enhanced message formatted for the particular device/application on which the enhanced message will be displayed….” EX1004, 6:5-9.
PNG
media_image7.png
564
900
media_image7.png
Greyscale
Gura, Fig. 1b
The input devices (e.g., mobile telephone 110.1) constitute the first user device, and the output devices (e.g., cellular telephone 120.1) constitute the second user device. EX1003 ¶¶141-146. A POSITA would know that the input devices for sending text messages, instant messages (IMs), emails, etc. and output devices for displaying enhanced text messages, enhanced IMs, enhanced emails, etc. present[] a…user interface. Id. ¶¶141-146.
Thus, Gura discloses both a first user device presenting a first user interface and a second user device presenting a second user interface as recited in claim 1.
a messaging cloud including:
Gura’s enhanced messaging system includes a messaging cloud (i.e., text message enhancement system implemented in a messaging network). EX1003
PNG
media_image8.png
975
1153
media_image8.png
Greyscale
¶¶150-155. It is a system for enhanced messaging that is remotely located in a network such as the Internet (i.e., cloud). EX1004, 5:17-19, 5:21-25, 6:12-15; EX1003 ¶151. Figure 2 illustrates an embodiment corresponding to the messaging cloud (i.e., text message enhancement apparatus connected with the SMSC and MMSC in the cellular network). Id.
Gura, Fig. 2 (annotated)
As described below, Gura discloses the required elements of a messaging cloud:
at least one database of rich media data and semantic model configured to provide certain functions recited in the claim (see Sections VII.B.5-VII.B.10); and
a messaging service configured to provide certain functions recited in the claim (see Sections VII.B.11-VII.B.15).
at least one database of rich media data, and a semantic model, the semantic model configured to:
Gura discloses a database of rich media data. EX1003 ¶¶157-165. For example, Figure 2 illustrates an “image database 30,” which “stores images for insertion into the message.” EX1004, 7:16.
PNG
media_image9.png
790
1349
media_image9.png
Greyscale
Gura, Fig. 2 (annotated)
Gura’s database includes multimedia items such as photographs, video clips, music and other sound clips, animations, and advertisements, all for insertion into enhanced messages. Id., 1:9-11, 4:32-5:1, 6 (first full paragraph), 7:25-27, 9:25-26, 11:20, 12:9, 13:3-7, 15:12, 17:12-13, 25:22-26, 27:1-3, 27:20, 27:26, 28:7-8, 31:12-14, 35 (claim 18); EX1003 ¶159. The ’149 Patent refers to such media items as “Rich Media Data.” EX1001, 1:41-42 (“images, photographs[,] audio and video (Rich Media Data)”), 1:49-50 (“rich media content (for example, images, photographs, audio, video, etc.)”). Thus, Gura discloses at least one database of rich media data. EX1003 ¶159.
In addition, Gura discloses a semantic model, which (as discussed above) refers to a “a manner of organizing a database based, at least in part, on the meaning of objects in the database.” See Section V.B, above. Gura’s image database is indexed in a manner that associates keywords, key phrases, and the message’s context with images based, at least in part, on meaning—just as in the ’149 Patent. EX1003 ¶160. For example, the character string “:-)” is associated with a smiling face because they both represent (i.e., they mean) a smile or happiness, just as in the ’149 Patent. EX1001, 9:40. Similarly, the keyword “love” is associated with an image of hearts that has the same meaning: love. EX1003 ¶160. The ’149 Patent makes the same association between the shortcut “(heart)” and the emotion “INLOVE.” EX1001, 10:48.
Gura creates these associations by “indexing” the image database. Gura’s text message enhancement apparatus employs “[a] message interpreter 24” that “receives the SMS message and analyzes it for suitable enhancements” including by “analyzing the text of the message being sent for keywords or key phrases, based on which images may be selected” and also “the context of the message, say the identity of the recipient,” which “may be matched with pre-set lists of the sender.” EX1004, 6:15-21. Gura’s image database is “categorized in ways that are appropriate for use with the message interpreter” meaning that images are “indexed for usability with the message interpreter.” Id., 7:16-20. “According to the indexing used in the image database 30, the text of the original message is correlated with images in the image database.” Id., 7:28-29. Thus, the “indexing used in the image database” corresponds to the semantic model recited in claim 1 because it is a “a manner of organizing a database based, at least in part, on the meaning of objects in the database.” EX1003 ¶162.3
Further, as described below, the image database “indexing” (i.e., semantic model) provides the four functions required by the claim: (1) categorize a personalized image library based on contact data and emotional state data; (2) receive a contact data input from a first device; (3) receive an emotional state data input from the same device; and (4) automatically select an image for insertion into a message…based on the received contact data and received emotional state data. EX1001, 11:35-46; EX1003 ¶164; see Sections VII.B.6-VII.B.9.
categorize a personalized image library based on contact data and emotional state data;
Gura discloses this limitation. EX1003 ¶¶167-173. Gura’s image database contains an image library because it holds various types of images (e.g., pictures, videos, etc.). See, e.g., EX1004, 4:32-5:1, 7:25-27, 31:12-14, 35 (claim 18).
The image library in Gura’s image database is also personalized. EX1003 ¶¶168-170. Gura’s user is “provided with a personalized web site to manage his accounts, set up new preferences for new recipients etc.” EX1004, 9:20-21. Using the personalized web site, “[t]he user can set his preferences at any given time for images/video/music he likes to be added to his text” and “can tag files on the site and index them with his preferences of text and language.” Id., 9:24-28. In particular, Gura describes a “Pictures Administration & Provisioning Interface” used for “[a]dding and delet[ing] pictures” and “[a]dding Key words to the files” in the database. Id., 13:3-7; see also id., 11:16-17, 15:1-2, 15:14-17, 25:22-26, 27:1-3, 28:7-8. Thus, users can create individualized sets of images for insertion, including images for particular recipients (e.g., location, etc.) and images based on sender information (e.g., emotion). EX1003 ¶169. By using an individualized set of images, the user’s “simple text message is personalized by automatically including content reflecting the text meaning, based on an analysis of the message text, context, and user preferences.” EX1004, 5:25-27; see also id., 6:21-24 (“Thus certain images may be used for groups of users, say business contacts, and other images may be used for personal contacts. Yet again other images or sets of images may be used for designated close personal contacts.”).
Gura further discloses that the personalized image library in the image database can be categorize[d]…based on contact data and emotional state data. EX1003 ¶¶171-172. Gura states that “[t]he image database is preferably a large database in which the images are selected and categorized in ways that are appropriate for use with the message interpreter.” EX1004, 7:16-17. Gura describes searching for images based on the “context” of the message, including the “identity of the recipient” (i.e., contact data), and “keywords,” which can indicate an emotional state of the user (i.e., emotional state data). EX1004, 2:3-4, 2:11-12, 6:18-19 (“keywords indicating sentiment may be used to select images illustrating the corresponding emotion”), 6:20-21 (“context” includes “the identity of the recipient”). Indeed, Gura’s “image database comprises images grouped into categories” and the “context [e.g., contact data] determines a category in which to search” and the “key words or phrases [e.g., emotional state data] identify which images within said determined category.” Id., 34 (claim 11); see also id., 7:2-3 (“The context indicates a set or collection of images to be used and the keywords point to individual images within the set or collection.”).
Thus, Gura’s “indexing” used in the image database (i.e., semantic model) provides for categoriz[ing] a personalized image library based on contact data and emotional state data, as recited in claim 1. EX1003 ¶173.
receive a contact data input from the first user device;
Gura discloses this limitation. EX1003 ¶¶175-177. In Gura, a “message interpreter 24 receives the SMS message” from the first cellular telephone (i.e., first user device) and analyzes it for keywords and the “context of the message,” which includes “the identity of the recipient” (i.e., contact data). EX1004, 6:15-21. The keywords and context can then be used to search for an image based on the particular “indexing” of the image database (i.e., semantic model). Id., 7:2-3 (“The context indicates a set or collection of images to be used and the keywords point to individual images within the set or collection), 7:28-29 (“According to the indexing used in the image database 30, the text of the original message is correlated with images in the image database.”), 34 (claim 11); EX1003 ¶176. Thus, Gura’s “indexing” used in the image database (i.e., semantic model) receive[s] a contact data input from the first user device. Id. ¶177.
receive an emotional state data input from the first user device; and
Gura discloses this limitation. EX1003 ¶¶179-182. Gura explains that “[t]ext message users are well known for using emoticons, short combinations of punctuation marks that indicate emotion in a message,” for example, “the colon, dash, close parenthesis emoticon :-) indicating happiness.” EX1004, 1:15-17; see also EX1001, 9:38-10:60 (identifying similar combinations of punctuation marks indicating “emotions”). And Gura’s enhanced messaging apparatus converts such emoticons and other “keywords indicating sentiment…to select images illustrating the corresponding emotion,” as shown in Figure 1a. EX1004, 6:18-19.4
PNG
media_image10.png
580
750
media_image10.png
Greyscale
Gura, Fig. 1 (annotated)
As discussed in the prior section, the keywords (which can include emotional state data) and context (which can include contact data) are then used to search for an image based on the particular “indexing” of the image database. Id., 7:2-3, 7:28-29, 34 (claim 11); EX1003 ¶180. Thus, Gura’s “indexing” used in the image database (i.e., semantic model) receive[s] an emotional state data input from the first user device, as recited in claim 1. Id. ¶182.
automatically select an image for insertion into a message from a plurality of images in the personalized image library based on the received contact data and received emotional state data, and
Gura discloses this limitation. EX1003 ¶¶184-190. Specifically, a “simple text message is personalized by automatically including content reflecting the text meaning, based on an analysis of the message text, context, and user preferences.” EX1004, 5:25-27.5 This is illustrated in Figure 1a.
PNG
media_image11.png
580
1014
media_image11.png
Greyscale
Gura, Fig. 1a (annotated)
The text includes emotional state data (i.e., “:-)” and “love”) and the message’s “context” includes contact data (e.g., the identity or location of the recipient). EX1003 ¶188. Images are selected for insertion based on the received contact data (i.e., context including recipient identity or location) and received emotional state data (i.e., keywords indicating sentiment or emotion). EX1004, 7:2-3 (“The context indicates a set or collection of images to be used and the keywords point to individual images within the set or collection.”), 34 (claim 11). Thus, Gura’s “indexing” used in the image database (i.e., semantic model) provides for automatically select[ing] an image for insertion into a message from a plurality of images in the personalized image library based on the received contact data and received emotional state data. EX1003 ¶190.
wherein the database is a Media Messaging Platform,
A POSITA would understand this “wherein” clause to mean that the recited database is part of a Media Messaging Platform, not that the database is itself a messaging platform. EX1003 ¶104. Both parties read the claim in this way.
EX1022, 38 (Patent Owner asserting with regard to this limitation that “the accused software system, Confluence, provides for a Media Messaging Platform between users…in conjunction with the database of rich media data.”). This understanding is also consistent with the specification, which describes the database as one component of the Media Messaging Platform (MMP). EX1001, 5:14-26, Fig. 1; see also id., 5:27-35, Fig. 2 (MMP Photocons Library and Database).
PNG
media_image12.png
1064
1328
media_image12.png
Greyscale
’149 Patent, Fig. 1 (annotated)
In contrast, the specification never discloses, or even suggests, a database that is itself a messaging platform or has any messaging functionality. EX1003 ¶104. Indeed, during prosecution the applicant added this “wherein” clause and argued that it distinguished the cited prior art because the reference did not disclose “a Media Messaging Platform database” (i.e., a database in a Media Messaging Platform). EX1002, 364; EX1003 ¶104.
PNG
media_image13.png
914
1346
media_image13.png
Greyscale
Gura discloses a database that is part of a Media Messaging Platform, as required by this limitation properly understood. EX1003 ¶¶192-195. As discussed above, the term Media Messaging Platform, as used in the ’149 Patent, means “a cloud-based platform for modifying messaging or texting services by personalizing messages or texts with multimedia inserts.” See Section V.A, above. Gura’s text message enhancement apparatus 100, for example, constitutes such a Media Messaging Platform. Id. ¶194.
Gura, Fig. 1b (annotated)
Gura’s text message enhancement apparatus is a “cloud-based platform” because it interoperates with numerous systems, such as mobile telephone, web, email, and IM systems (see, e.g., EX1004, Fig. 1b), is “preferably located in the cellular network” (id., 6:10-15) or “other networks, such as the Internet” (id., 5:23-25), and includes a database of multimedia items as illustrated below (id., 7:16-27, 10:29-30, Fig. 2).
PNG
media_image14.png
815
1081
media_image14.png
Greyscale
Gura, Fig. 2 (annotated)
Gura’s apparatus retrieves images and other media from the image database and inserts them in the message to create an enhanced message. Id., 4:30-5:1, 7:16-24, Fig. 1a. Thus, Gura’s text message enhancement apparatus corresponds to the Media Messaging Platform and discloses this limitation of claim 1. EX1003 ¶195.
a messaging service configured to:
Gura’s enhanced messaging system includes a messaging service. EX1003 ¶¶197-201. In Gura’s Figure 2, the text message enhancement apparatus receives SMS text messages that are enhanced with media items and sent to the recipient as MMS messages. EX1004, 5:17-19, 7:21-24, Fig. 2. Thus, Gura’s enhanced messaging system includes a messaging service that enhances SMS messages for delivery as MMS messages. EX1003 ¶199. Gura also teaches enhancing “instant messaging” and “email messages” with media items. EX1004, 5:23-6:4.
Further, Gura’s messaging service provides the four functions required by the claim: (1) receive a message…including a rich media data request from the first user device; (2) retrieve the requested rich media data from the database; (3) insert the retrieved rich media data into the message while maintaining text flow; and (4) present the message to the recipient. EX1001, 11:49-56; see Sections VII.B.12-VII.B.15, below. Thus, Gura discloses the claimed messaging service. EX1003 ¶¶200-201.
receive a message, the message including a rich media data request from the first user device,
PNG
media_image15.png
806
648
media_image15.png
Greyscale
Gura discloses this limitation. EX1003 ¶¶203-206. Gura’s message grabber 34 identifies and grabs messages sent by system subscribers (e.g., first cellular telephone 12 in Figure 1a). EX1004, 8:3-4, Fig. 2. The message grabber directs the text message to the message interpreter 24 where the text is analyzed for suitable enhancement. Id., 8:3-6, 8:20-30, Fig. 2. Figure 3 illustrates these initial steps performed by Gura (steps 1 and 2). Id., 8:22-25.
Gura, Fig. 3 (annotated)
The text message enhancement apparatus analyzes the message for keywords, phrases, and context and then uses that information to identify and retrieve corresponding media items to insert in the message. Id., 6:15-7:3, 34 (claim 11). Thus, the keywords, key phrases, and context, alone or in combination, constitute rich media data request[s] that are included in the message. EX1003 ¶205. In the Figure 1a example, the punctuation marks “:-)” in text message 10 act to request a rich media replacement from Gura’s system—constituting a rich media data request received from the first user device—and are converted to an image of a smiley face in enhanced message 20. EX1004, 5:17-19, Fig. 1a. Similarly, the word “love” indicates to Gura to insert an image of hearts and is thus a rich media data request. Id. Thus, Gura discloses a messaging service that receive[s] a message, the message including a rich media data request from the first user device. EX1003 ¶206.
retrieve the requested rich media data from the at least one database of rich media data,
Gura discloses this limitation. EX1003 ¶¶208-210. As explained in the prior section, Gura’s enhanced messaging system uses keywords, key phrases, and the message’s context to identify and retrieve media items in the image database to enhance the received text message. EX1004, 6:15-7:3, 34 (claim 11), Fig. 2. This step is illustrated in Figure 3 (step 3).
PNG
media_image16.png
846
679
media_image16.png
Greyscale
Gura, Fig. 3 (annotated)
Gura’s enhanced messaging system “search[es] a database of media files” based on the analyzed text message and “insert[es] media files retrieved by said search into said analyzed message….” Id., 2:11-16. Thus, Gura discloses a messaging service that retrieve[s] the requested rich media data from the at least one database of rich media data. EX1003 ¶210.
insert the retrieved rich media data into the message while maintaining text flow, and
Gura discloses this limitation. EX1003 ¶¶211-213. Once media items are retrieved from the database, Gura’s enhanced messaging system inserts the selected media items into the message to create an enhanced message. EX1004, 5:17-20. This step is illustrated in Figure 3 (step 4).
PNG
media_image17.png
583
468
media_image17.png
Greyscale
Gura, Fig. 3 (annotated)
As Gura explains, “[t]he database is connected to an image inserter 32 which takes the images selected at the database and inserts them into the text messages.” Id., 7:21-22. Thus, Gura discloses a messaging service that insert[s] the retrieved rich media data into the message. EX1003 ¶211.
Further, Gura discloses inserting the media items into the message while maintaining text flow. EX1003 ¶211. As illustrated in Figure 1a, for example, the image of a smiley face is inserted in enhanced message 20 to replace the punctuation marks “:-)” in the original text message 10 while maintaining the text flow. Id.6
PNG
media_image18.png
580
1016
media_image18.png
Greyscale
Gura, Fig. 1a (annotated)
In the example, Gura automatically replaces the character string “:-)” with an image located where the characters were typed in the message thereby maintaining text flow. Id. Thus, Gura discloses a messaging service that insert[s] the retrieved rich media data into the message while maintaining text flow. Id., ¶213.
present the message to the second user device via the second user interface; and
Gura discloses this limitation. EX1003 ¶¶215-218. After Gura retrieves the media items from the database and inserts them into the text message, “[t]he result is a multimedia message which is an enhanced version of the original text message,” and “[t]he multimedia message is then sent to the recipient.” EX1004, 7:21-24. This is illustrated in Figure 3 (step 7).
PNG
media_image19.png
581
468
media_image19.png
Greyscale
Gura, Fig. 3 (annotated)
In the Figure 1a example, the enhanced, multimedia message is sent to the second cellular telephone where it is displayed as enhanced message 20. Id., 5:17-19. Thus, Gura discloses a messaging service that present[s] the message to the second user device via the second user interface. EX1003 ¶218.
a network operably coupling the first user device, the second user device, and the messaging cloud,
Gura discloses this limitation. EX1003 ¶¶220-224. Gura’s messaging cloud (i.e., text message enhancement apparatus, SMSC, and MMSC in the Fig. 2 embodiment) resides in the network connecting the message sender (i.e., a first user device) and the message receiver (i.e., a second user device). See, e.g., EX1004, Abstract, 4:30-5:1, 5:17-19, 5:23-25; 6:10-15; EX1003 ¶223. The first user device sends a message over the cellular network; it is received and modified by Gura’s messaging cloud to include media items; and then it is sent over the cellular network to the second user device as an enhanced message. See, e.g., EX1004, 5:19-22, 10:29-30. Thus, in Gura, the network is operably coupling the first user device, the second user device, and the messaging cloud. EX1003 ¶224.
wherein the message is originally formatted on the first user device via the first user interface such that the rich media data request is within the message.
Gura discloses this limitation. EX1003 ¶¶226-229. Figure 1a provides an example in which the message (i.e., text message 10) is originally formatted on the first user device via the first user interface. Id. ¶¶227-228.
PNG
media_image20.png
729
877
media_image20.png
Greyscale
Gura, Fig. 1a
Further, the message is formatted such that the rich media data request is within the message. Id. ¶228. For example, text message 10 includes text strings (i.e., “:-)” and “love”) that are converted to media items inserted into enhanced message 20 (i.e., the smiley face and hearts). Id.7 Those text strings are examples of rich media data request[s] because they indicate to Gura’s text message enhancement apparatus that they should be replaced by media items (i.e., rich media data). Id. ¶¶203-205, 228. Thus, Gura discloses that the message is originally formatted on the first user device via the first user interface such that the rich media data request is within the message. Id. ¶229.
Claim 2
Claim 2 adds the additional limitation:
The communications system of claim 1, wherein the messaging cloud further comprises a parser configured to analyze the message for a shortcut key, the shortcut key representative of a request for rich media data.
Gura discloses this additional limitation. EX1003 ¶¶231-234. In Gura, a message interpreter “receives the SMS message and analyzes it for suitable enhancements.” EX1004, 6:15-16, Fig. 2 (Message Interpreter 24). “The analysis may involve analyzing the text of the message being sent for keywords or key phrases” that can be used to select media items for insertion in the enhanced message. Id., 6:16-19. Gura’s keywords correspond to the multi-character shortcut key[s] in the ’149 Patent. See, e.g., EX1001, 9:38-10:60 (Tables 1 and 2 with corresponding emotions and shortcuts); EX1003 ¶¶232, 234.8 In the embodiment illustrated in Figure 2, the “message interpreter 24” includes “text analyzer 26,” which “is preferably able to analyze text, including abbreviations often used in text messaging, such a ‘gr8’ for ‘great.’” EX1004, 7:7-9. Gura confirms that “[s]uch text analyzers are known in the art” and are “intended to allow machine analysis of the text, and may include text recognition and natural language processing (NLP) algorithms as appropriate.” Id., 7:8-11. Thus, Gura’s messaging cloud includes a parser (i.e., message interpreter with text analyzer) that analyze[s] the message for a shortcut key that is representative of a request for rich media data. EX1003 ¶¶233-234.
Claim 3
Claim 3 adds the additional limitation:
The communications system of claim 1, wherein the rich media data request is included within the message by at least one of tapping, clicking, dragging, or typing of a shortcut key on the first user interface.
Gura discloses this additional limitation. EX1003 ¶¶236-238. In the Figure 1a example, the user has prepared text message 10 using cellular telephone 12. EX1004, 5:17, Fig. 1a. A POSITA would know that the user either typed the message using the telephone’s keypad or its touch screen such that the shortcut key (e.g., “:-)”) was typed on the first user interface. EX1003 ¶237.9 Thus, Gura discloses, teaches, and suggests to a POSITA that the rich media data request is included within the message by…typing of a shortcut key on the first user interface. Id. ¶238.
Claim 4
Claim 4 adds the additional limitation:
The communications system of claim 1, wherein the message is at least one of an email message, text message, Internet-based instant message, or other network-transmitted message.
Gura discloses this additional limitation. EX1003 ¶¶240-242. Gura’s Figure 1a illustrates an exemplary text message 10 (i.e., text message) that is enhanced for delivery over a cellular network. EX1004, 5:15-22, Fig. 1a. Gura also states that “the same enhancement may be carried out using other networks” and identifies examples of such text messages including “Application-to-
application messages (i.e., instant messaging)” (i.e., Internet-based instant message), “Email messages” (i.e., an email message), and “Application-to-mobile telephone messages (i.e. from the operator website to the user mobile telephone)” (i.e., other network-transmitted message). Id., 5:23-6:4, Fig. 1b. Thus, Gura discloses that the message is at least one of an email message, text message, Internet-based instant message, or other network-transmitted message. EX1003
¶242.
Claim 5
Claim 5 adds the additional limitation:
The communications system of claim 4, wherein the first user device and the second user device are respectively at least one of a mobile phone, a pager, a personal digital assistant (PDA), a tablet, a laptop computer, or a desktop computer.
Gura discloses this additional limitation. EX1003 ¶¶244-246. Gura’s Figure 1a illustrates an exemplary text message 10 sent by a first cellular telephone 12 (i.e., a mobile phone) for delivery as enhanced message 12 to a second cellular telephone (i.e., a mobile phone). EX1004, 5:15-22, Fig. 1a. Thus, Gura discloses that the first user device and the second user device are respectively…a mobile phone…. EX1003 ¶246.
Claim 6
Claim 6 adds the additional limitation:
The communications system of claim 1, further comprising a messaging notification subsystem configured to evaluate a status of second user device and present the message to the second user device when the second user device is online.
Gura discloses this additional limitation. EX1003 ¶¶248-252. In Figure 6, Gura provides a “flow chart of a positive scenario, in which an incoming SMS is received, the application converts the SMS to an MMS message, and the MMSC succeeds in delivering the MMS to the recipient.” EX1004, 16:15-17. The figure illustrates the flow and signaling beginning with the “User Sender” transmitting the text message via the “SMSC” (on the left side of the figure) to the delivery from the “MMSC” to the “User Recipient” (on the right side of the figure). Id., Fig. 6; EX1003 ¶249. A POSITA would know that Gura is describing the then-existing sequence of signals in MMS systems for “deferred retrieval” of a message. Id.; EX1006, 315-316, Fig. 6.14. The signaling exchange between the MMSC and the recipient corresponds to, and discloses, the limitations recited in claim 6. EX1003 ¶249.
Once Gura’s system has enhanced the message to include multimedia items, the enhanced message is sent to the MMSC (Multimedia Service Center) for delivery to the recipient. EX1004, 16:15-17. “The MMSC tries to deliver the message” by sending “the end-user an MM1_Notification_Request.” Id., 18:10-11; EX1007, 315-316, Fig. 6.14.
PNG
media_image21.png
1070
992
media_image21.png
Greyscale
Gura, Fig. 6 (excerpt, annotated)
If the user recipient’s device is online, it sends an MM1_Notification_Response back to the MMSC, as illustrated. EX1003 ¶251; EX1007, 315-316, Fig. 6.14. Then the recipient’s device signals for delivery of the enhanced message by sending an MM1_Retrieve_Request to the MMSC. EX1003 ¶251; EX1007, 315-316, Fig. 6.14. This exchange of signals informs the MMSC that the recipient device is online and ready to receive the enhanced message. EX1003 ¶251; EX1007, 314. Thus, Gura’s text message enhancement apparatus works with the MMSC to provide a messaging notification subsystem configured to evaluate a status of the second user device (i.e., confirming it is online and prepared to receive the message).
Having confirmed that the recipient device is online and ready to receive the enhanced message (i.e., the second user device is online), the MMSC delivers the enhanced message. EX1003 ¶252.
PNG
media_image22.png
1070
983
media_image22.png
Greyscale
Gura, Fig. 6 (excerpt, annotated)
This is illustrated in Figure 6 as the MM1_Retrieve_Response sent by the MMSC in response to the user device’s MM1_Retrieve_Request (i.e., the user recipient sends a request to retrieve the enhanced message and the MMSC sends the enhanced message in response). Id. ¶252; EX1007, Fig. 6.14.
Thus, Gura’s text message enhancement apparatus employs the MMSC to provide a messaging notification subsystem configured to evaluate a status of the second user device (i.e., confirming it is online and prepared to receive the message) and then present[s] the message to the second user device when the second user device is online (i.e., sending the enhanced message once requested by signaling from the user recipient device). EX1003 ¶252.
Claim 7
Claim 7 adds the additional limitation:
The communications system of claim 6, wherein the messaging notification subsystem is further configured to push the message to the second user device when the second user device is offline.
Gura discloses this additional limitation. EX1003 ¶¶254-258. Gura describes the message flow and signaling when enhanced message delivery fails. EX1004, 4:18-19, Fig. 8. Gura states that the “MMSC tries to deliver the message” and “in case of failure” returns a failure code such as “End-User rejected the request, Time-Out, etc.” Id., 18:10-13. A POSITA would recognize that one reason the MMSC cannot deliver the enhanced message is if the recipient device is not connected to the MMS system or is not an MMS device (i.e., it is offline). EX1003 ¶256. Gura recognizes this, stating that in some cases the enhanced message “could not be delivered to the destination as an MMS message” “[t]ypically because the recipient is not sufficiently powerful to receive MMS.” EX1004, 19:17-19.10 In other words, the message cannot be delivered to a recipient device that is unable to receive multimedia items (i.e., it is offline). EX1003 ¶256. In this case, the MMSC returns an error code causing Gura’s system to send the message “on to the recipient as an unenhanced message.” EX1004, 19:9-19, 23:29-32; see also id., 18:30-32 (“In the event of an error, the engine fetches the originating message…and sends it to the destination as an unenhanced SMS message, as discussed below.”).
This SMS fallback functionality is illustrated in Gura’s Figure 8 showing message flow and handling when the MMSC fails to deliver the enhanced message. EX1004, 16:21-23.11
PNG
media_image23.png
954
1170
media_image23.png
Greyscale
Gura, Fig. 8 (annotated)
By sending the unenhanced text message to the recipient (i.e., Outgoing SMS Message), the system is push[ing] the message to the second user device because the message is sent to that device without any affirmative request or other action by the recipient, as illustrated in Figure 8. EX1003 ¶258. Thus, Gura discloses that the messaging notification subsystem is further configured to push the message to the second user device when the second user device is offline. Id.
Claim 8
Claim 8 adds the additional limitation:
The communications system of claim 1, further comprising an image library organizer configured to: upload the rich media data to the at least one database of rich media data; and assign shortcut keys to rich media data.
Gura discloses this additional limitation. EX1003 ¶¶262-266. Gura discloses a “Picture Administration & Provisioning Interface” that accesses the image database (i.e., database of rich media data) and is used for “[a]dding…pictures” (i.e., upload[ing] the rich media data) and “[a]dding Key words to the files” (i.e., assign[ing] shortcut keys to rich media data).12 EX1004, 13:3-5; see also id., 13:6-7 (“Picture Administration & Provisioning is done through a web client who prepares the data for quick searching by the application.”), 15:14-17 (Picture Administration & Provisioning Interface accesses the database), 25:23-26 (Section entitled “Pictures Maintenance” stating: “The message enhancement server and product preferably provides a tool to works with the database…to insert a new picture into the database or to add key words for searching for pictures” and that “also allows for maintenance of the current pictures.”). Thus, Gura discloses an image library organizer (i.e., Picture Administration & Provisioning Interface) configured to: upload the rich media data to the at least one database of rich media data and to assign shortcut keys to rich media data. EX1003 ¶266.
Claim 9
Claim 9 adds the additional limitation:
The communications system of claim 1, wherein each of the plurality of images has a relation to a shortcut key.
Gura discloses this additional limitation. EX1003 ¶¶268-269. In the example illustrated in Figure 1a, enhanced message 20 includes an image of a smiley face and an image of hearts (i.e., plurality of images) and each of those images is correlated with a keyword (i.e., shortcut key) in the original text message 10. EX1004, 5:15-20, Fig. 1a. The smiley face image is related to the text string “:-)” and the image of hearts is related to the text string “love.” Thus, in the example each of the plurality of images has a relation to a shortcut key, as recited in claim 9. EX1003 ¶269. To the extent Gura’s image database contains additional media items not illustrated in the figure, a POSITA would know that each is associated with a shortcut key so that it can be inserted into an enhanced message—that is the express purpose of Gura’s text message enhancement apparatus (i.e., associating keywords with images for use creating enhanced messages). Id.; EX1004, 2:1-3:3, 4:30-5:2, 5:15-20.
Claim 10
A method of streamlined media insertion on an electronic device, the method comprising:
Gura discloses a method of streamlined media insertion on an electronic device as discussed in claim 1. See Section VII.B, above. The electronic device creates and sends the message, corresponding to the first user device of claim 1 (e.g., Gura’s first cellular telephone 12). EX1003 ¶271. Gura provides streamlined media insertion because the system is designed to use text in a message to automatically identify media items for insertion without requiring the user to search for and attach those items while typing the message. Id. ¶272.
initiating, by the electronic device, a messaging session with a receiving user device;
Gura discloses this limitation. EX1003 ¶¶275-276. Gura’s Figure 1a example illustrates a first cellular telephone 12 (i.e., the electronic device) sending a text message through the cellular network for delivery (i.e., initiat[es]…a messaging session) to the second cellular telephone (i.e., a receiving user device). EX1004, 5:15-22, Fig. 1a, Fig. 1b (messaging between Mobile telephone (SMS) 110.1 and Cellular telephone (MMS) 122.1); EX1003 ¶275. Gura also describes embodiments for enhancing text messages in instant messaging systems. EX1004, 5:23-31, Fig. 1b (communications between IM application 110.5 and IM application 120.5). A POSITA would know that instant messaging systems establish a messaging session between the user devices. EX1003 ¶276.
receiving a message on the electronic device, the message including at least one media insertion request;
Gura discloses this limitation. EX1003 ¶¶278-282. In Figure 1a, the user has prepared text message 10 using cellular telephone 12 (i.e., the electronic device). EX1004, Fig. 1a. The figure therefore illustrates an example of receiving a message on the electronic device (i.e., it was typed on the first cellular telephone by the user). EX1003 ¶279. Further, the exemplary message includes keywords (i.e., “:-)” and “love”) that Gura’s enhanced messaging system will replace with media items inserted to form enhanced message 20. EX1004, 5:15-20, Fig. 1a. Thus, the keywords constitute media insertion requests contained in the message. EX1003 ¶280.
evaluating the at least one media insertion request at a messaging cloud, the messaging cloud including a Media Messaging Platform;
Gura discloses a messaging cloud as discussed in claim 1. See Section VII.B.4, above. The system receives an SMS text message and analyzes the text to identify keywords, key phrases, and the context of the message that can be used to identify media items for insertion in an enhanced message. EX1004, 6:16-22. Thus, Gura’s messaging cloud is evaluating the at least one media insertion request. EX1003 ¶285.
Further, Gura’s enhanced messaging system (i.e., messaging cloud) includes a Media Messaging Platform. Gura’s text message enhancement apparatus 100 constitutes a Media Messaging Platform as discussed in claim 1. See Section VII.B.10, above. The apparatus connected to, and operating with, the SMSC and MMSC of the cellular network constitutes the messaging cloud. See Section VII.B.4, above; EX1003 ¶288-291. The two claimed elements are identified below using Gura’s Figure 2 as a reference. Id. ¶¶290-291.13
PNG
media_image24.png
564
591
media_image24.png
Greyscale
PNG
media_image25.png
660
686
media_image25.png
Greyscale
Gura, Fig. 2 (annotated)
Thus, Gura discloses evaluating the at least one media insertion request at a messaging cloud and that the messaging cloud include[es] a Media Messaging Platform. EX1003 ¶¶285-292.
retrieving media from at least one Media Messaging Platform database of media based on the evaluated at least one media insertion request, said database of media including a personalized image library and a semantic model, the semantic model configured to:
Gura discloses this limitation. EX1003 ¶¶294-298. Gura analyzes text messages for keywords, key phrases, and the context of the message, and uses that information to search for media items in the image database to insert in the enhanced message sent to the recipient. EX1004, 6:16-22, Fig. 2. The keywords, key phrases, and the message context, alone or in combination, constitute media insertion requests because they inform the system that a media item should be inserted in the enhanced message. EX1003 ¶297. Thus, Gura discloses retrieving media from at least one Media Messaging Platform database of media (i.e., image database 30) based on the evaluated at least one media insertion request. EX1003 ¶297. In addition, Gura’s image database includes a personalized image library and a semantic model, as in claim 1. See Sections VII.B.5-VII.B.6, above.
Claim 10 further states that the semantic model is configured to provide the same four functions recited in claim 1. EX1001, 11:37-46, 12:43-52. The semantic model and each of the recited functions is similarly disclosed by Gura as in claim 1 (see Sections VII.B.5-VII.B.9, above) and as further clarified in the following sections (see Sections VII.K.6-VII.K.9, below).
categorize the personalized image library based on contact data and emotional state data;
Gura discloses this limitation as in the virtually-identical limitation of claim 1. See Section VII.B.6, above.
receive a contact data input from the electronic device;
Gura discloses this limitation as in the similar limitation of claim 1. See Section VII.B.7, above. The only difference between the corresponding limitations is that in claim 10 the contact data is received from the electronic device, while in claim 1 it is received from the first user device. EX1001, 11:49-50, 12:46. Under either name, Gura discloses the claimed device. EX1003 ¶304. For example, “first cellular telephone 12” in Figure 1a corresponds to, and therefore discloses, the electronic device of claim 10 and also the first user device of claim 1. Id.
receive an emotional state data input from the electronic device; and
Gura discloses this limitation as in the similar limitation in claim 1. See Section VII.B.8, above. Again, the only difference in the corresponding limitations is the term electronic device in claim 10 and first user device in claim 1. Gura discloses the claimed device. EX1003 ¶307.
automatically select an image for insertion into the message from the plurality of images in the personalized image library based on the received contact data and received emotional state data, and
Gura discloses this limitation as in the virtually-identical limitation of claim 1. See Section VII.B.9, above.
inserting the retrieved media into the message while maintaining text flow;
As discussed in connection with claim 1 (see Section VII.B.14, above), Gura insert[s] the retrieved media into the message while maintaining text flow, as illustrated in Figure 1a.
PNG
media_image27.png
581
1051
media_image27.png
Greyscale
Gura, Fig. 1a (annotated)
Thus, Gura discloses this limitation of claim 10. EX1003 ¶¶311-312.
presenting the message to the receiving user device via a receiving user interface,
As discussed in connection with claim 1 (see Section VII.B.15, above), after Gura retrieves media items from the database and inserts them into the message, “[t]he result is a multimedia message which is an enhanced version of the original text message” and “[t]he multimedia message is then sent to the recipient.” EX1004, 7:21-24. In Gura’s Figure 1a, the enhanced message is sent to the second cellular telephone and presented to the recipient on the telephone’s user interface. EX1003 ¶¶315-316. Thus, Gura discloses presenting the message to the receiving user device via a receiving user interface. Id. ¶316.
wherein a network operably couples the electronic device, the receiving user device, and the messaging cloud, and
As discussed in connection with claim 1 (see Section VII.B.16, above), Gura’s messaging cloud resides in the network connecting the message sender (i.e., the electronic device) and the enhanced message receiver (i.e., the receiving user device). EX1004, Abstract, 4:30-5:1, 5:17-19, 5:23-25, 6:10-15. The user of the electronic device sends a message that is modified by Gura to include media items and is then delivered to the user of the receiving user device as an enhanced message. Id., 5:19-22. Thus, the cellular network (in an embodiment) is a network [that] operably couples the electronic device, the receiving user device, and the messaging cloud. EX1003 ¶¶318-321.
the message session formatted on the electronic device such that the rich media data request is within the message.
As discussed in connection with claim 1 (see Section VII.B.17, above), Figure 1a provides an example in which the message session formatted on the electronic device (as illustrated by text message 10) is such that the rich media data request is within the message. EX1003 ¶324. Text message 10 includes text strings (i.e., “:-)” and “love”) that constitute rich media data request[s] because they indicate to Gura’s system that they should be replaced by media items (i.e., rich media data). Id. And they are within the message. Id.
Claim 11
Claim 11 recites a limitation similar to claim 2 and is similarly disclosed by Gura. See Section VII.C, above. For example, Figure 1a illustrates a text message containing a string of text (i.e., text message 10) received from the user on the electronic device (i.e., cellular telephone 12). EX1004, Fig. 1a. In the illustrated example, the string of text (i.e., text message 10) includes shortcut key[s] (i.e., “:-)” and “love”). When the message is sent, Gura’s message interpreter “receives the SMS message and analyzes it for suitable enhancements.” Id., 6:15-16, Fig. 2. Gura uses the shortcut key[s] to identify rich media data (i.e., the smiley face and hearts) for insertion in enhanced message 20. Id., Fig. 1a. Thus, Gura discloses that receiving the message on the electronic device comprises receiving a string of text (i.e., text message 10) and that the string of text includ[es] a shortcut key as the at least one media insertion request (i.e., keyword such as “love” in Figure 1a). EX1003 ¶329.
Claim 12
Claim 12 recites a limitation similar to claim 3 and is similarly disclosed by Gura. See Section VII.D, above. In Gura’s Figure 1a example, the user has prepared text message 10—including the shortcut key[s] “:-)” and “love”—using cellular telephone 12 (i.e., the electronic device). EX1004, Fig. 1a. A POSITA would know that the user typed the message, including shortcut key[s], using either the keypad or touch screen on the electronic device. EX1003 ¶333. Thus, Gura discloses, teaches, and suggests to a POSITA that the at least one media insertion request is included within the message by…typing of a shortcut key on the electronic device, as claimed. Id. ¶¶333-334.
Claim 13
Claim 13 recites a limitation that is virtually identical to claim 4 and is similarly disclosed by Gura. See Section VII.E, above.
THE CHALLENGED CLAIMS ARE OBVIOUS OVER GURA ALONE (GROUND 1), WITH JENSEN (GROUND 2), OR WITH FIFIELD (GROUND 3)
Gura discloses the claimed semantic model because it describes indexing the image database to organize data objects based, at least in part, on their meaning and to provide the four functions recited in claims 1 and 10. See Section VII.B.5-VII.B.9 and VII.K.5-VII.K.9, above. Should Patent Owner argue that the claimed semantic model is more narrowly limited to the type of data structure disclosed in Figure 3 of the ’149 Patent (i.e., a multidimensional lookup table), the challenged claims are obvious nevertheless because such data structures were well-known and would have been an obvious implementation detail (Ground 1). EX1003 ¶¶339-340; EX1005, 7-32, Fig. 2.5; EX1006, [0003], [0049], Fig. 5; EX1013, 3:20-35, 11:12-15, 27:4-8, Fig. 13. Indeed, Gura suggests employing a multidimensional lookup table by stating that “images [are] grouped into categories” (e.g., separate tables) and then the “context” (including recipient identity and location, i.e., contact data) “determines the category in which to search and said key words or phrases” (including those that indicate sentiment and emotion, i.e., emotional state data) “identify which images within said determined category” to select. EX1004, 34 (claim 11).
In addition, the combination of Gura with either Jensen (Ground 2) or Fifield (Ground 3) would also render the claims obvious as discussed below.
Gura Combined with Jensen
Jensen discloses a “cube” data structure that is “a truly multidimensional data structure for capturing and analyzing data.” EX1005, 7. Jensen explains that although the term “cube” suggests three dimensions, it “can have any number of dimensions.” Id. The dimensions are “used for two purposes: the selection of data and the grouping of data at a desired level of detail.” Id., 9. For example, Jensen illustrates a multi-dimensional “cube” data structure regarding book sales data. Id., 19.
PNG
media_image28.png
733
804
media_image28.png
Greyscale
Jensen, Fig. 2.5
Thus, Jensen’s cube is a data structure used to organize and identify data items in a database, and it is the same type of data structure illustrated in Figure 3 of the ’149 Patent. EX1003 ¶343.
The cube data structure, as in Jensen, was well-known long before the ’149 Patent. EX1013, 3:20-35, 11:12-15, 27:4-8, Fig. 13; EX1005, Abstract. A POSITA implementing Gura’s messaging system, including the image database, would have naturally looked to a book such as Jensen for fundamental concepts regarding database organization and implementation. EX1003 ¶344; EX1005, xi (“The object of the book is to offer university students, researchers, and practitioners a comprehensive, integrated, and understandable coverage of multidimensional data models and data modeling concepts.”). It would have been an obvious choice to organize Gura’s media items into categories to be searched based on context (including contact data) and keywords (including emotional state data). EX1003 ¶345; EX1004, 34 (claim 11). A POSITA would have recognized the benefits of employing Jensen’s cube because it provides simple organization and efficient searching of data items. EX1003 ¶345; EX1005, vi (book covers “fundamental concepts” including “data cubes and their elements” and noting they were “well-known in the database community”). Further, because the cube data structure was well-understood and rudimentary by 2013, it would have been well within the skill of a POSITA to implement in Gura’s system with a high expectation of success and no unreasonable experimentation. EX1003 ¶345.
Thus, the combination of Gura with Jensen renders the challenged claims obvious even under an unreasonably narrow reading of semantic model limiting it to the type of data structure illustrated in Figure 3 of the ’149 Patent. EX1003 ¶346.
Gura Combined with Fifield
Fifield describes generating a “semantic model” composed of multiple tables each containing multiple fields, as in the ’149 Patent. EX1003 ¶347. In particular, Fifield allows developers to “generate a semantic model from the application development process, as opposed to generating it as a second development effort after deployment of the application.” EX1006 ¶[0006]. Fifield’s database schema are used to store application metadata, and a “semantic model” is created containing “data such as information concerning the relationships between the stored application metadata, information about how the stored application metadata is analyzed,…and information about how to navigate the data.” Id. Fifield’s “semantic model in effect places a layer on top of the business application metadata so that it can be properly understood, navigated, analyzed, etc.” Id. “Semantic models also refer to items using ‘friendly names’ rather than by their possibly cryptic and/or hard to understand real names.” Id.
According to Fifield, prior art database were designed first and then “a separate effort is undertaken to create an additional set of metadata that describes a ‘user friendly’ business model on that same schema” (i.e., a semantic model). EX1006 ¶[0004]. In contrast, Fifield’s “semantic model generator 226” generates a semantic model without need for a separate development process. Id. ¶¶[0006], [0029], Fig. 2.
PNG
media_image29.png
993
1425
media_image29.png
Greyscale
Fifield, Fig. 2 (highlighted)
The “semantic model generator component 226” in the illustrated application development system “generates a business intelligence semantic model 230….” Id. ¶[0029]. This can “occur as a result of developer 202 or user 220 creating and/or modifying application metadata.” Id.
Fifield’s “semantic model 230” is illustrated in Figure 5. EX1006 ¶[0049].
PNG
media_image30.png
486
1088
media_image30.png
Greyscale
Fifield, Fig. 5
It includes “multiple tables 320-1 through 320-N” with “each table including one or more fields (for example fields 230-1 through 320-M shown in table 320-1).” Id. A POSITA would recognize this data structure provides the same multidimensional lookup functionality as in Figure 3 of the ’149 Patent. EX1003 ¶349; EX1005, 12 (multidimensional data structures can have “one dimension table for each dimension”). Thus, Gura combined with Fifield discloses a semantic model that performs the claimed functions (Gura) by employing a multidimensional lookup (Fifield). EX1003 ¶349.
Fifield is analogous art because it addresses the same problem of organizing information in a database to provide association between objects based on their meaning. EX1006 ¶[0040] (“When creating the semantic model 230, the semantic model generator 226 uses heuristics about what various settings in the application metadata actually mean to end users….”); EX1001, 9:28-10:60, Fig. 10, Fig. 14 (shortcuts associated with emotions/images based on meaning); EX1004, Fig. 1 (keywords associated with images with same meaning). While Gura suggests a multidimensional lookup (EX1004, 34 (claim 11)), Fifield discloses such a semantic model explicitly. EX1003 ¶350.
A POSITA would recognize that Gura’s image database indexing could be implemented using Fifield’s application development environment with “semantic model generator,” which creates a semantic model including multiple tables each containing multiple fields (i.e., providing a multidimensional lookup as illustrated in Figure 3 of the ’149 Patent). EX1006 ¶¶[0029], [0049]. A POSITA would recognize the benefits of such a combination. For example, as Fifield states, integrating development of the semantic model with application development “potentially increases the quality and decreases the time-to-market for [the] overall solution.” Id. ¶[0014]. Further, the semantic model creates a “user friendly” interface between end-user and database (id. ¶¶[0003]-[0004], which would improve user interactions with Gura’s image database. EX1003 ¶351. Also, Fifield explains that if a later version of the underlying application is deployed, the semantic model generator can automatically update the semantic model. EX1006 ¶¶[0057], [0062]. Such benefits identified by the references themselves would motivate a POSITA to combine Gura with the teachings of Fifield to disclose the claimed invention. EX1003 ¶351.
A POSITA could have combined the references as discussed. Fifield’s system, including the semantic model generator, can be implemented in software (EX1006 ¶[0018]) and would have been well within a POSITA’s capabilities by 2013. EX1003 ¶352. Further, Fifield’s semantic model of Figure 5 would have been a simple data structure for a POSITA to develop using even rudimentary software tools and languages in wide use by 2013. Id. A POSITA could have combined Gura with Fifield’s semantic model generator and semantic model with a reasonable expectation of success and without unreasonable experimentation. Id.
Thus, the combination of Gura with Fifield renders the challenged claims obvious even under an unreasonably narrow reading of semantic model limiting it to the type of data structure illustrated in Figure 3 of the ’149 Patent. EX1003 ¶353.
CLAIM 7 IS RENDERED OBVIOUS BASED ADDITIONALLY ON LE BODIC (GROUNDS 4, 5, AND 6)
In the related litigation, Patent Owner contends that claim 7 covers a messaging feature that sends an offline recipient an email with a “view comment” link letting the recipient access the message including its multimedia items. EX1022, 73-76. As discussed above, Gura describes a SMS fallback functionality that sends the original SMS text message when the enhanced message fails to deliver; for example, if the recipient is not connected to the system or is using a non-MMS legacy device (i.e., the second user device is offline). See Section VII.H, above. Gura does not disclose a link in the SMS fallback message, but that functionality was available in then-existing SMS/MMS networks as explained by Le Bodic. EX1003 ¶354. Le Bodic states that “if a user sends a multimedia message (via MMS) to a legacy user” (i.e., a user with “a legacy phone without support for MMS”), “the network can generate a short message and deliver it (via SMS) to the legacy user” providing “the address of an Internet page showing the message contents that can be viewed by the legacy user on any Web browser….” EX1007, 216. Thus, Gura’s enhanced messaging system combined with Le Bodic’s teachings of an SMS message containing an Internet address to retrieve the enhanced message discloses claim 7.
A POSITA implementing Gura’s enhanced messaging system would have naturally looked to a textbook like Le Bodic describing then-existing SMS and MMS networks. EX1003 ¶355; EX1007, back cover. Gura’s disclosures are in the context of such networks. See, e.g., EX1004, 5:15-22, Figs. 2, 6, 8. Thus, a POSITA would have looked to Le Bodic for further details on then-existing architecture, functionality, and messaging protocols in SMS/MMS networks. EX1003 ¶355.
A POSITA also would have recognized the benefits of implementing Gura’s system to permit then-existing delivery functionality to legacy devices because “[a]ny operator providing the MMS service must ensure that special functions are in place to handle multimedia messages sent to legacy phones.” EX1007, 232. A POSITA would have recognized the benefit of sending Gura’s SMS fallback text message modified to include a link permitting the recipient to view the enhanced message, not merely the original text message. EX1003 ¶356. Further, a POSITA would be motivated to take advantage of existing SMS/MMS functionality and could have done so with a high expectation of success because the functionality was well-known and standardized before the ’149 Patent. Id.
1 With the exception of Virtanen and Winkler, the prior art citied is also listed on the IDS filed by Applicant on 25 November 2024.
2 For ease of review, claim language is presented in italics.
3 To the extent Patent Owner argues that “semantic model” implies a semantic analysis of the text message to divine its meaning—by using natural language processing, for example—Gura discloses such an analysis. EX1004, 7:7-12 (“The text analyzer is preferably able to analyze text…. Principally the feature is intended to allow machine analysis of the text, and may include text recognition and natural language processing (NLP) algorithms as appropriate. The algorithms used are optimized to understand context from very short messages and are preferably self-learning.”). Under this narrower construction of semantic model, Gura’s message interpreter (including text analyzer) analysis and database indexing would correspond to the claimed semantic model. EX1003 ¶163.
4 Gura’s enhanced messaging system can also use the recipient’s “location” to identify media items for insertion. EX1004, 6:25-28. The recipient’s “location” (e.g., home address) constitutes contact data. EX1003 ¶176.
5 Gura also discloses an optional embodiment in which the user can preview and approve images before sending. EX1004, 8:29-30; EX1003 ¶189.
6 Should Patent Owner argue that the retrieved rich media data must be a photograph, as opposed to the smiley face and hearts in Figure 1a, Gura discloses this limitation anyway because Gura’s image database stores photographs and videos (among other things) for insertion in enhanced messages. EX1004, 1:9-11, 4:32-5:1, 6 (first full paragraph), 7:25-27, 9:25-26, 11:20, 12:9, 13:3-7, 15:12, 17:12-13, 25:22-26, 27:1-3, 27:20, 27:26, 28:7-8, 31:12-14, 35 (claim 18).
7 Gura also states that “[t]he message always contains data about the…destination,” confirming that the message includes information about its “context” such as the identity or location of the recipient (i.e., contact data). EX1004, 16:28-30. Thus, Gura discloses contact data being within the message. EX1003 ¶228.
8 Gura’s keywords also correspond to a rich media data request (e.g., emotional state data) in the message. See Section VII.B.12, above. Claim 2 confirms that a shortcut key can be a request for rich media data, and Gura’s keywords disclose both. EX1003 ¶232.
9 Should Patent Owner argue that the phrase typing of a shortcut key on the first user interface limits the claim to use of a touch screen, Gura renders such a limitation obvious as it is one of two alternatives (i.e., using the keypad or the touch screen), both of which would be readily apparent to a POSITA based on Gura’s Figure 1a and related text. EX1003 ¶237.
10 A POSITA would understand Gura’s reference to a recipient “not sufficiently powerful to receive MMS” as a legacy device not configured to receive and display such multimedia messages. EX1003 ¶256.
11 In Figure 8, the message is rejected by the recipient. A POSITA would recognize from Gura that the same SMS fallback procedure would result if the delivery failed because the recipient device is not connected to the MMSC or is a non-MMS legacy device (i.e., it is offline). EX1003 ¶257.
12 Gura’s key words correspond to shortcut keys of claim 8 because they are associated with (i.e., assigned to) images in the image database. EX1003 ¶266.
13 Gura’s “message grabber 34” (illustrated in Figure 2) “may be located on the network or may be a client installed at the sender’s telephone.” EX1004, 8:3-6. Therefore, it may or may not be part of the Media Messaging Platform or messaging cloud depending on the particular implementation. EX1003 ¶290.