DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 21-37 are pending.
Claims 1-20 are canceled.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 21, 23-26, 28-31 and 33-35 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-18 of U.S. Patent No. US11941368. Although the claims at issue are not identical, they are not patentably distinct from each other because the limitations in the above indicated claims of the instant application are anticipated by the respective claimed limitations in the above indicated claims of U.S. Patent No. US11941368. See the claim anticipation mapping below.
Instant application
Claims
Patent US11941368
Claims
21
1
22
none
23
1
24
1, 3
25
1, 9
26
11
27
none
28
11
29
11, 13
30
11, 9
31
11
32
none
33
11
34
1, 13
35
1, 9
36
none
37
none
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 21-37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rogowski et al (WO2014162211) in view of Bean et al (US20160259608).
Regarding claims 21, 26 and 31, Bean teaches an electronic device, comprising: a camera; a display; one or more processors, the one or more processors configured to:
(Rogowski, "a computer system for translating a foreign language on a mobile camera device, the system comprising: a mobile camera for capturing a video image of the one or more words in the first language for translation of the first language text; a program code; a processor for processing the program code; one or more memories connected to the processor", [0024]; "using camera elements and software of the smartphone to focus on printed object text in one language, which text may then be seen in a display of the smartphone", [0049]; an electronic device (smartphone/mobile camera device) comprising a camera, a display, and one or more processors configured to execute program code)
display, via the display, a preview image captured by the camera;
(Rogowski, "positioning the mobile camera device to display a video image of one or more words in the first language which need to be translated so that the mobile camera device can capture frames of a video feed of the one or more words in the first language for translation", [0011]; "which text may then be seen in a display of the smartphone", [0049]; displaying a real-time preview image (video feed) captured by the camera via the device's display)
identify an image quality metric of the preview image and motion information of the electronic device;
(Rogowski, "comparing information quality in the current frame of the language translation video image to the information quality in the previous frame of the language translation video image", [0011]; "tracking is performed on the current frame and the previous frame, to obtain vectors with x and y movements. As users will usually focus on text on a flat area, processing determines if a user is holding the phone steadily by checking if the vectors are consistent and small.", [0105]; identifying an image quality metric (comparing information quality between frames) and identifying motion information (checking x and y movement vectors to see if the device is held steadily).
Rogowski, Fig. 1; "In step 112, a determination is made whether or not pre-processing has revealed a text precursor which would indicate there is some preliminary text information suggestive of a text character.", [0093]; "proceeds to step 124 where a determination is made as to whether the process string is blank, meaning no language character for translation has been found in the process string", [0094]; "Tracking is performed on the current frame and the previous frame, to obtain vectors with x and y movements. As users will usually focus on text on a flat area, processing determines if a user is holding the phone steadily by checking if the vectors are consistent and small.", [0105]; Fig. 3, "Processing at step 316 first determines a) whether or not the vectors of the tracking locations are similar. Processing at step 316 also determines (b) whether or not the average vectors of the tracking locations accumulated between recognition result updates are small. In addition, processing at step 316 determines (c) whether or not the current image size matches the image size of the previous frame", [0110]; Rogowski teaches identifying an image quality metric of the preview image (cropped frame) by determining if pre-processing reveals a "text precursor" ([0093]) and if the "process string is blank" (no language character found) ([0094]) before translation. Rogowski also teaches identifying motion information by tracking frames to obtain "vectors with x and y movements" and determining if "vectors are consistent and small" ([0105]), specifically checking if vectors "are similar", "average vectors... are small", and "current image size matches... previous frame" [0110]. Together Rogowski teaches identifying both an image quality metric (text precursor/character presence) of the preview image and motion information (tracking vector consistency/size) of the electronic device)
determine whether an object depicted in the preview image captured by the camera has text written on the object, wherein the text is written in a language;
(Rogowski, "focus on printed object text in one language", [0049]; "determination is made whether or not pre-processing has revealed a text precursor which would indicate there is some preliminary text information suggestive of a text character.", [0093]; determining if an object has text written on it in a specific language by looking for preliminary text information suggestive of text characters)
when the object has text written on the object, the image quality metric of the preview image being above a reference, and the motion information indicates motion of the electronic device being within a reference range, requesting translation of the text written on the object to text of another language;
(Rogowski, "a determination is made as to whether the process string is blank, meaning no language character for translation has been found in the process string and the process proceeds to step 126, where a message is displayed to the user "Image unclear,"", [0094]; "Thus, the current frame is processed forward to step 352 as illustrated in FIG. 3B only if the process determines that: (1) the vectors of the tracking algorithm are consistently similar; (2) the average vectors of the tracked locations accumulated between recognition result updates are small;", [0110]; "process uses results from an optical character recognition (OCR) process for translating the characters of Simplified Chinese to English words.", [0095]; ensuring an image is clear/not blank (quality metric above a reference) and ensuring motion vectors are consistently similar and small (motion within a reference range), and upon these conditions, requesting translation of the detected language text to another language.
Rogowski, Fig. 1; "If a text precursor has been identified in step 112”, [0094], ... “If the determination at step 124 indicates a character has been found, then the process proceeds to step 130... In step 132, the process uses results from an optical character recognition (OCR) process for translating the characters of Simplified Chinese to English words", [0095]; "the current frame is processed forward to step 352 as illustrated in FIG. 3B only if the process determines that: (1) the vectors of the tracking algorithm are consistently similar; (2) the average vectors of the tracked locations accumulated between recognition result updates are small; and (3) image size of the current frame matches the image size of the previous frame.", [0110]; "The process proceeds to step 368 where the focused and cropped multi-line text frame proceeds to a multi-line text recognition process which starts at step 402 in FIG 4”, [0111], ... “The translation process in step 416 translates the Chinese characters, the text of the first language, into English words, the text of the second language", [0115]; Rogowski teaches that when text is detected ("text precursor identified" [0094], "character has been found" [0095], image quality metric above reference), the single-line process proceeds to variant conversion (step 130) and requests translation via OCR (step 132) [0095]. Rogowski teaches that for multi-line AR translation, the current frame proceeds to recognition/translation (steps 352, 368, 416) only if motion information is within a reference range: vectors "consistently similar", "average vectors... small", and "image size... matches" [0110]. Together Rogowski teaches requesting translation when the object has text, the preview image quality metric is met (character found), and motion information indicates steadiness (vectors consistent/small, size match))
when the motion information indicates that the motion of the electronic device is within the reference range, replacing the text written on the object with the text of the another language; and
(Rogowski, "If the vectors are consistent and small, then character recognition can be performed", [0105]; "The English translation 1202 is displayed inside the target box in this embodiment of the present invention with each line of Chinese characters and its English translation overlap. In one embodiment, the English translations are on top of the Chinese characters.", [0140]; replacing the original text by displaying the translated text augmented reality overlay "on top of" the original foreign text, provided the motion vectors indicate the device is held steadily within the reference range.
Rogowski teaches requesting translation (performing OCR/translation steps) only when both the preview image quality metric is satisfied and motion information is within a reference range)
when the motion information indicates that the motion of the electronic device is outside the reference range, maintaining the text written on the object in the language.
(Rogowski, "When processing at step 316 indicates based on determinations (a-c) that there has been significant movement of current frame relative to the previous frame, then processing at step 316 sends both the current and previous frames of multi-line text to step 318. At step 318 the process deletes the current and previous frames", [0110]; Bean, “If the radial speed of the object 112 in the augmented reality display 106 is greater than the threshold speed for the information associated with the object 112, then the information in the overlay may be moving off the screen too fast for the user 104 to comfortably focus on ... Processing continues to step 218 where the information associated with the object 112 is filtered out and is not displayed in the overlay.", [0036]; While Rogowski implicitly maintains the original text by dropping frames and cancelling the translation overlay when significant movement is detected, Bean explicitly teaches filtering out augmented reality overlays so that they are not displayed when movement/speed exceeds a threshold (motion outside the reference range), effectively maintaining the visual presentation of the original object and its native text without the augmented translation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Bean into the system or method of Rogowski in order to prevent visual clutter and distraction when the device is moving too fast for the user to comfortably read the augmented reality translation overlays, by explicitly filtering out the translated text overlay and maintaining the clean view of the original object text when motion exceeds a reference threshold. The combination of Rogowski and Bean also teaches other enhanced capabilities.
Regarding claims 22, 27 and 32, the combination of Rogowski and Bean teaches its/their respective base claim(s).
The combination further teaches the electronic device of claim 21, wherein the another language is user selectable.
(Rogowski, “the second language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic”, [claim 7])
Regarding claims 23, 28 and 33, the combination of Rogowski and Bean teaches its/their respective base claim(s).
The combination further teaches the electronic device of claim 21, further comprising:
a communication module configured to transmit the text written on the object and receive the text of the another language from a server.
(Rogowski, “a digital device enabled to practice the invention may open a data connection to an on-line server, and some functionality may be provided by software and data at the on-line serve”, [0049]; e.g., the language translation function which requires significant resources may be off-loaded to the remote server as a predicted option)
Regarding claims 24, 29 and 34, the combination of Rogowski and Bean teaches its/their respective base claim(s).
The combination further teaches the electronic device of claim 21, wherein translating the text written on the object further comprises:
extracting partial images comprising pixels of the preview image containing the text and pixels of the preview image that surround the text.
(Rogowski, the Chinese menu image in the background of Fig. 13 is portion of the Chinese menu of Fig. 8)
Regarding claims 25, 30 and 35, the combination of Rogowski and Bean teaches its/their respective base claim(s).
The combination further teaches the electronic device of claim 24, wherein replacing the text written on the object with the text of the another language comprises:
wherein the text written on the object moves from a first position to a second position, and
wherein the pixels of the preview image are at the first position, displaying the text of the another language at the second position.
(Claim interpretation: the pixels of the preview image contain the text written on the object in the 1st language; when the object moves from 1st position to a 2nd position, it means that the text pixels of the 1st language move from 1st position to a 2nd position; the text of the another language (the 2nd language) will be displayed to be following the movement of the text pixels on the object of the preview image;
Bean, Fig. 1; “Similarly, real world object 112 is represented in the overlay by a marker 122 to which any information associated with the real world object 112 is anchored. As real world object 112 moves in the direction shown by arrow 130, the marker 122 moves with it, so that the marker's position continues to correspond with the position of the real world object 112 as seen by the user 104”, [0023], marker 122 can be the overlay information on a moving object when the object speed is, e.g., less than a threshold (Fig. 2, S212, S214); in this case, the position of the marker (overlay information) will follow the position of the object (e.g., the Chinese character image) due to the object movement since the object movement in the field of view of the user is caused by the user camera movement; the overlay information can be the English text translated from the Chinese text image of Rogowski)
Regarding claim 36, the combination of Rogowski and Bean teaches its/their respective base claim(s).
The combination further teaches the electronic device of claim 21, wherein the image quality metric of the preview image comprises a blur state of the preview image, a number of feature points included in the preview image, or a distribution of the feature points included in the preview image.
(Rogowski, "where the vector results from the tracking are used to prevent processing unstable images with motion blurring.", [0110]; "Sometimes the camera may have thought it is already focused, so it will process blurred image without trying to focus. Accordingly, the process provides a tap-to-focus functionality for users to force it to re-focus.", [0092]; "during image capture of the foreign text using a hand-held mobile camera device such as a smart phone, there are image capture problems which include camera movement, poor text image focus, and improper foreign text illumination. OCR requires a clear distinctive text image for an accurate and stable foreign text translation", [0007]; the image quality metric includes evaluating a blur state of the preview image; analyzing the image to prevent the processing of "unstable images with motion blurring" and checking if the system is attempting to process a "blurred image" due to poor focus)
Regarding claim 37, the combination of Rogowski and Bean teaches its/their respective base claim(s).
The combination further teaches the electronic device of claim 21, further comprising
a communication module,
(Rogowski, "may open a data connection to an on-line server, and some functionality may be provided by software and data at the on-line server", [0049]; "a computer system for translating a foreign language on a mobile camera device, the system comprising: a mobile camera for capturing a video image of the one or more words in the first language for translation of the first language text; a program code; a processor for processing the program code; one or more memories connected to the processor", [0024]; Rogowski teaches an electronic device (mobile camera device/smartphone) that opens a data connection to an on-line server, which inherently requires a communication module (e.g., wireless transceiver) to establish the data connection for transmitting/receiving translation data)
wherein the one or more processors are further configured to, based on the request for translation:
identify at least one text region of the preview image based on at least one of a history of using a translation service or a distribution of feature points included in the preview image;
(Rogowski, "In other embodiments a digital device enabled to practice the invention may open a data connection to an on-line server, and some functionality may be provided by software and data at the on-line server", [0049]; "The process then proceeds to step 508 where a connected component analysis is performed on the horizontally blurred image to partition the binarized image of the process string into its segments", [0119]; "The features set can extract the relevant information from the input data and perform satisfactorily in the algorithm of the present invention to detect and isolate various features of a video stream", [0088]; "counting a number of similar language text characters in a current language text translation image string and in a previous language translation image string; and (b) calculating what fraction of these similar language text characters are overlapping in the current and the previous language translation image strings", [0022]; Rogowski teaches network communication to server, text region identification via connected components/horizontal blurring, feature extraction for feature distribution, and frame-to-frame character overlap comparison (history))
extract, from the preview image, at least one partial image corresponding to the at least one text region of the preview image;
(Rogowski, "connected component analysis with cropping of the corresponding regions in the original image", [0120]; "In step 108, the process crops the image from the image processing bounding box selected by the user. To crop the image refers to removal of the outer parts of the image of the first language characters in a bounding box to accentuate the characters", [0093]; cropping partial images for each text region (connected component regions, bounding box))
transmit, through the communication module to at least one external electronic device, the at least one partial image, an entire image corresponding to the preview image, and language information for translating text included in the at least one partial image or the entire image; and
(Rogowski, "In other embodiments a digital device enabled to practice the invention may open a data connection to an on-line server, and some functionality may be provided by software and data at the on-line server", [0049]; "Furthermore, the hardware may include an interface with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), a wireless network, and/or the Internet among others) to permit the communication of information with other computers coupled to the networks", [0146]; "Another embodiment of the present invention also includes a method for translating a first language selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic", [0013]; transmitting data to server via network interface for translation, with language selection (language info))
receive, through the communication module from the at least one external electronic device, the text of the another language and region information indicating a position, a size, and a length of the text of the another language.
(Rogowski, "In other embodiments a digital device enabled to practice the invention may open a data connection to an on-line server, and some functionality may be provided by software and data at the on-line server", [0049]; "Processing in step 362 checks each individual line of text separately in a frame of the multi-line text image; checks to see that text is staying in the bounding box", [0112]; "the process of step 354 proceeds to step 360 to adjust the text location from the previous frame of the translated text appearing on the image display of the mobile camera device. The adjustment uses the vectors calculation process for comparing the current frame to the previous frame and moves the translated text appearing on the image display of the mobile camera device", [0113]; receiving translations from server and tracking text position/size/length via bounding box and vectors for AR overlay)
Response to Arguments
Applicant's arguments filed on 7/16/2026 with respect to one or more of the pending claims have been fully considered but they are not persuasive.
Regarding claim(s) 21, 26 and 31, Applicant, in the remarks, argues that the combination of the cited reference(s) fails to teach the newly amended limitations in the claims.
The Examiner respectfully disagreed. The office action has been updated to address applicant’s argument. See the updated review comments for details.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 9/19/2026
Prosecution Insights