DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1, 5, 10, 13, 15 and 16 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 12 – 14, 16 and 17 of U.S. Patent No. 12,375,620.
Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1, 5, 10, 13, 15 and 16 of the instant application are broader and fully encompassed by claims 12 – 14, 16 and 17 of ‘620.
Instant Claims
12,375,620 Claims
Differences
A method comprising:
receiving a user selection of an object in a live image stream displayed in a camera viewfinder graphical user interface at a client device;
removing the object from the live image stream;
displaying, in the camera viewfinder graphical user interface, the live image stream with the object removed despite a camera of the client device capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
12. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: displaying in a camera viewfinder graphical user interface, at a client device, a live image stream from a camera of the client device; detecting a plurality of objects in the live image stream;
receiving, via the camera viewfinder graphical user interface, a selection of an object of the plurality of objects in the live image stream;
removing the object in response to the selection; generating content and filling a hole corresponding to the removed object in the live image stream with the content; and
displaying the live image stream with the content in place of the removed object in the camera viewfinder graphical user interface despite the camera capturing the object in the live image stream.
13. The non-transitory computer-readable medium of claim 12, wherein the operations further comprise capturing a digital image from the live image stream with the object removed.
Instant claim 1 is broader and all encompassing of claims 12 and 13 of ‘620.
5. The method of claim 1, further comprising:
generating by a content aware fill machine learning model, content to fill a hole created by removal of the object; and filling, the hole in the live image stream with the content.
12. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: displaying in a camera viewfinder graphical user interface, at a client device, a live image stream from a camera of the client device; detecting a plurality of objects in the live image stream;
receiving, via the camera viewfinder graphical user interface, a selection of an object of the plurality of objects in the live image stream;
removing the object in response to the selection; generating content and filling a hole corresponding to the removed object in the live image stream with the content; and
displaying the live image stream with the content in place of the removed object in the camera viewfinder graphical user interface despite the camera capturing the object in the live image stream.
Claim 12 teaches the exact same limitations as claim 5.
10. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
displaying, in a camera viewfinder graphical user interface at a client device, a live image stream being captured by a camera of the client device;
detecting one or more objects in the live image stream;
removing an object of the detected one or more objects from the live image stream;
displaying, in the camera viewfinder graphical user interface, the live image stream with the object removed despite a camera of the client device capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
12. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
displaying in a camera viewfinder graphical user interface, at a client device, a live image stream from a camera of the client device;
detecting a plurality of objects in the live image stream;
receiving, via the camera viewfinder graphical user interface, a selection of an object of the plurality of objects in the live image stream;
removing the object in response to the selection; generating content and filling a hole corresponding to the removed object in the live image stream with the content; and
displaying the live image stream with the content in place of the removed object in the camera viewfinder graphical user interface despite the camera capturing the object in the live image stream.
13. The non-transitory computer-readable medium of claim 12, wherein the operations further comprise capturing a digital image from the live image stream with the object removed.
Instant claim 10 is broader and all encompassing of claims 12 and 13 of ‘620.
13. The non-transitory computer-readable medium of claim 10, wherein:
detecting that the object is moving in the live image stream at a speed greater than a speed threshold; and removing the object of the detected one or more objects from the live image stream based on the object exceeding the speed threshold.
17. The non-transitory computer-readable medium of claim 12, wherein: detecting comprises detecting the plurality of objects in a sequence of frames of the live image stream utilizing an object detection neural network; receiving the selection comprises determining an object speed threshold; and removing comprises removing a moving object from the live image stream when the moving object exceeds the object speed threshold.
Claim 13 is broader and all encompassing of claim 17.
15. The non-transitory computer-readable medium of claim 10, wherein the operations further comprise
generating content utilizing a content aware fill neural network to to replace the object in the live image stream.
16. The non-transitory computer-readable medium of claim 15, wherein generating the content comprises: determining a location of the movable icon; and utilizing a content aware fill neural network to generate updated content based on the selected content within the movable icon.
Claim 15 is broader and all encompassing of claim the combination of 12, 15 and 16.
16. A system comprising: at least one memory device; and at least one processor configured to cause the system to perform operations comprising:
receiving a user selection of an object in a live image stream displayed in a camera viewfinder graphical user interface at a client device;
removing the object from the live image stream;
displaying, in the camera viewfinder graphical user interface, the live image stream with the object removed despite a camera of the client device capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
12. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
displaying in a camera viewfinder graphical user interface, at a client device, a live image stream from a camera of the client device;
detecting a plurality of objects in the live image stream;
receiving, via the camera viewfinder graphical user interface, a selection of an object of the plurality of objects in the live image stream;
removing the object in response to the selection; generating content and filling a hole corresponding to the removed object in the live image stream with the content; and
displaying the live image stream with the content in place of the removed object in the camera viewfinder graphical user interface despite the camera capturing the object in the live image stream.
13. The non-transitory computer-readable medium of claim 12, wherein the operations further comprise capturing a digital image from the live image stream with the object removed.
Instant claim 16 is broader and all encompassing of claims 12 and 13 of ‘620.
Claim 10 – 12, 14 and 15 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 2, 5 – 7 of U.S. Patent No. 12,375,620.
Instant Claims
12,375,620 Claims
Differences
10. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
displaying, in a camera viewfinder graphical user interface at a client device, a live image stream being captured by a camera of the client device;
detecting one or more objects in the live image stream;
removing an object of the detected one or more objects from the live image stream;
displaying, in the camera viewfinder graphical user interface, the live image stream with the object removed despite a camera of the client device capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
1. A computer-implemented method comprising:
displaying, in a camera viewfinder graphical user interface at a client device, a live image stream from a camera of the client device;
detecting a plurality of objects in the live image stream; selecting an object of the plurality of objects in the live image stream; removing the object from the live image stream;
displaying, in the camera viewfinder graphical user interface at the client device, the live image stream with the object removed despite the camera capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
Instant claim 10 is broader and all encompassing of claims 1 of ‘620.
11. The non-transitory computer-readable medium of claim 10, wherein detecting the one or more objects in the live image stream further comprises utilizing an object detection machine learning model to detect the one or more objects in frames of the live image stream and assign an object label to each of the one or more objects.
2. The computer-implemented method of claim 1,
wherein detecting the plurality of objects further comprises utilizing an object detection machine learning model to detect the plurality of objects in frames of the live image stream and assign an object label to each of the plurality of objects.
Instant claim 11 is broader and all encompassing of claims 1-2 of ‘620.
12. The non-transitory computer-readable medium of claim 11, wherein the operations further comprise:
determining a theme of the live image stream based on object labels of the one or more objects in the live image stream; selecting the object as unwanted based on the determined theme of the live image stream and an object label of the object; and removing the object of the detected one or more objects from the live image stream based on the object being unwanted.
5. The computer-implemented method of claim 2, wherein selecting the object of the plurality of objects in the live image stream comprises: determining a theme of the live image stream based on object labels of the plurality of objects in the live image stream; and selecting the object as unwanted based on the determined theme of the live image stream and the object label of the object.
Instant claim 12 is broader and all encompassing of claims 1-2, 5 of ‘620.
14. The non-transitory computer-readable medium of claim 10, wherein the operations further comprise: providing a selectable icon for each of the one or more objects detected in the live image stream; receiving a selection of a selectable icon for the object; and
removing the object of the detected one or more objects from the live image stream in response to receiving the selection of the selectable icon for the object.
6. The computer-implemented method of claim 1, wherein selecting the object further comprises: providing a selectable element in connection with display of the object in the live image stream; and receiving a selection of the selectable element.
Covered by claim 10
Instant claim 14 is broader and all encompassing of claims 1, 6 of ‘620.
15. The non-transitory computer-readable medium of claim 10, wherein the operations further comprise generating content utilizing a content aware fill neural network to replace the object in the live image stream.
7. The computer-implemented method of claim 1, further comprising: generating by a content aware fill machine learning model, content to fill a hole created by removal of the object; and filling, the hole in the live image stream with the content.
Instant claim 15 is broader and all encompassing of claims 1, 7 of ‘620.
Claims 1 – 2, 5 – 9, 16 – 19 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 5, 7 – 11 of U.S. Patent No. 12,375,620 in view of Jung (US 2019/0197673).
Regarding claim 1, ‘620 reads on the limitations as follows:
Instant Claims
12,375,620 Claims
Differences
A method comprising:
removing the object from the live image stream;
displaying, in the camera viewfinder graphical user interface, the live image stream with the object removed despite a camera of the client device capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
1. A computer-implemented method comprising: displaying, in a camera viewfinder graphical user interface at a client device, a live image stream from a camera of the client device; detecting a plurality of objects in the live image stream; selecting an object of the plurality of objects in the live image stream; removing the object from the live image stream; displaying, in the camera viewfinder graphical user interface at the client device, the live image stream with the object removed despite the camera capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
Instant claim 1 is broader and all encompassing of claim 1 of ‘620, except for the strike limitations.
‘620 fails to explicitly disclose receiving a user selection of an object in a live image stream displayed in a camera viewfinder graphical user interface at a client device.
In a similar field of endeavor, Jung teaches an imaging device for removing unwanted objects wherein a user 10 may select an object 2c included in the first image displayed on a touch screen of the image acquisition device 1000 (fig. 13; ¶114). In light of the teaching of Jung, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Jung’s teaching in ‘60s system because an artisan of ordinarily skill would recognize that this would help easily facilitate object selection by a user.
Regarding claim 2, ‘620 in view of Jung discloses the limitations of claim 1. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
The method of claim 1, further comprising:
tracking a location of the object in frames of the live image stream; removing the object from each of the frames of the live image stream based on the tracking of the object; and displaying the live image stream, in the camera viewfinder graphical user interface, with the object removed.
4. The computer-implemented method of claim 3, further comprising: tracking a location of the object in the frames of the live image stream; removing the object from each of the frames of the live image stream based on the tracking of the object; and displaying the live image stream, in the camera viewfinder graphical user interface, with the object removed.
Instant claim 2 is broader and all encompassing of claims 1-4 of ‘620
Regarding claim 5, ‘620 in view of Jung discloses the limitations of claim 1. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
5. The method of claim 1, further comprising: generating by a content aware fill machine learning model, content to fill a hole created by removal of the object; and filling, the hole in the live image stream with the content.
7. The computer-implemented method of claim 1, further comprising: generating by a content aware fill machine learning model, content to fill a hole created by removal of the object; and filling, the hole in the live image stream with the content.
Instant claim 5 is broader and all encompassing of claims 1 and 7 of ‘620
Regarding claim 6, ‘620 in view of Jung discloses the limitations of claim 5. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
6. The method of claim 5, further comprising:
receiving, prior to capturing the digital image, an indication from the client device that the content used to fill the hole is insufficient; receiving, prior to capturing the digital image, an identification of an area of the live image stream to inform the content aware fill machine learning model; generating, by the content aware fill machine learning model, updated content to fill the hole based on the area of the live image stream; and filling the hole in the live image stream with the updated content.
8. The computer-implemented method of claim 7, further comprising: receiving, prior to capturing the digital image, an indication from the client device that the content used to fill the hole is insufficient; receiving, prior to capturing the digital image, an identification of an area of the live image stream to inform the content aware fill machine learning model; generating, by the content aware fill machine learning model, updated content to fill the hole based on the area of the live image stream; and filling the hole in the live image stream with the updated content.
Instant claim 6 is broader and all encompassing of claims 1 and 7-8 of ‘620
Regarding claim 7, ‘620 in view of Jung discloses the limitations of claim 6. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
7. The method of claim 6, wherein receiving the indication from the client device further comprises:
providing a selectable icon in the camera viewfinder graphical user interface;
and receiving a selection of the selectable icon..
9. The computer-implemented method of claim 8, wherein receiving the indication from the client device further comprises: providing a selectable icon in the camera viewfinder graphical user interface; and receiving a selection of the selectable icon.
Instant claim 7 is broader and all encompassing of claims 1 and 7-9 of ‘620
Regarding claim 8, ‘620 in view of Jung discloses the limitations of claim 7. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
8. The method of claim 7, further comprising:
providing and displaying a movable element in response to the selection of the selectable icon; and receiving the identification of the area of the live image stream by detecting a location of the movable element.
10. The computer-implemented method of claim 9, further comprising: providing and displaying a movable element in response to the selection of the selectable icon; and receiving the identification of the area of the live image stream by detecting a location of the movable element.
Instant claim 7 is broader and all encompassing of claims 1 and 7-10 of ‘620
Regarding claim 9, ‘620 in view of Jung discloses the limitations of claim 8. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
9. The method of claim 8,
wherein receiving the identification of the area of the live image stream by detecting the location of the movable element comprises determining that the movable element is located on a portion of an updated live image stream being captured in response to panning the client device.
11. The computer-implemented method of claim 10, wherein receiving the identification of the area of the live image stream by detecting the location of the movable element comprises determining that the movable element is located on a portion of an updated live image stream being captured in response to panning the client device.
Instant claim 7 is broader and all encompassing of claims 1 and 7-11 of ‘620
Regarding claim 16, ‘620 reads on the limitations as follows:
Instant Claims
12,375,620 Claims
Differences
16. A system comprising: at least one memory device; and at least one processor configured to cause the system to perform operations comprising:
removing the object from the live image stream; displaying, in the camera viewfinder graphical user interface, the live image stream with the object removed despite a camera of the client device capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
1. A computer-implemented method comprising: displaying, in a camera viewfinder graphical user interface at a client device, a live image stream from a camera of the client device; detecting a plurality of objects in the live image stream; selecting an object of the plurality of objects in the live image stream; removing the object from the live image stream; displaying, in the camera viewfinder graphical user interface at the client device, the live image stream with the object removed despite the camera capturing the object in the live image stream; and
after removing the object from the live image stream, capturing by the client device a digital image from the live image stream with the object removed.
Instant claim 16 is broader and all encompassing of claim 1 of ‘620, except for the strike limitations.
‘620 fails to explicitly disclose receiving a user selection of an object in a live image stream displayed in a camera viewfinder graphical user interface at a client device.
In a similar field of endeavor, Jung teaches an imaging device for removing unwanted objects wherein a user 10 may select an object 2c included in the first image displayed on a touch screen of the image acquisition device 1000 (fig. 13; ¶114). In light of the teaching of Jung, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Jung’s teaching in ‘60s system because an artisan of ordinarily skill would recognize that this would help easily facilitate object selection by a user.
Regarding claim 17, ‘620 in view of Jung discloses the limitations of claim 16. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
17. The system of claim 16, wherein the operations further
utilizing an object detection machine learning model to detect one or more objects in frames of the live image stream and assign an object label to each of the one or more objects.
2. The computer-implemented method of claim 1, wherein detecting the plurality of objects further comprises utilizing an object detection machine learning model to detect the plurality of objects in frames of the live image stream and assign an object label to each of the plurality of objects.
Instant claim 17 is broader and all encompassing of claims 1-2 of ‘620
Regarding claim 18, ‘620 in view of Jung discloses the limitations of claim 17. ‘620 also teaches:
Instant Claims
12,375,620 Claims
Differences
18. The system of claim 17, wherein the operations further comprise:
determining that the object is unwanted based on an object label of the object; and removing the object from the live image stream based on the object being unwanted.
5. The computer-implemented method of claim 2, wherein selecting the object of the plurality of objects in the live image stream comprises: determining a theme of the live image stream based on object labels of the plurality of objects in the live image stream; and selecting the object as unwanted based on the determined theme of the live image stream and the object label of the object.
Instant claim 18 is broader and all encompassing of claims 1 and 5 of ‘620
Regarding claim 19, ‘620 in view of Jung discloses the limitations of claim 18. Jung also teaches wherein the operations further comprise: generating by a content aware fill machine learning model, content to fill a hole created by removal of the object; and filling, the hole in the live image stream with the content (fig. 18; ¶104,128)
Regarding claim 20, ‘620 in view of Jung discloses the limitations of claim 19. Jung also teaches wherein the operations further comprise displaying the live image stream, prior to capturing of the digital image, with the object removed and the generated content in place of the object as the live image stream changes over time or in response to movement of the client device capturing the live image stream (fig. 6; ¶98,127-128)
Claims 3 – 4 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 4 of U.S. Patent No. 12,375,620 in view of Jung (US 2019/0197673) in view of Ding (US 2018/0137892).
Regarding claim 3, ‘620 in view of Jung discloses the limitations of claim 1. The combination fails to explicitly disclose wherein tracking the location of the object in the frames of the live image stream comprises utilizing a similarity heuristic to determine similarity scores of pixels for additional objects in the frames of the live image stream that indicate how similar the additional objects are to the object.
In a similar field of endeavor, Ding teaches object tracking in videos, where the video tracking system utilizes a similarity heuristic to determine similarity scores for candidate objects in the video frames 102a-102n that indicate how similar the candidate objects are to the query object 110. In one or more embodiments, the video tracking system uses a spatially-constrained similarity measure with a voting map based measuring approach to identify the query object in the video frames 102a-102n (fig. 5; ¶44). In light of the teaching of Ding, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to use Ding’s teaching in ‘620s system because an artisan of ordinarily skill would recognize that this would facilitate object tracking across multiple frames.
Regarding claim 4, ‘620 in view of Jung in view of Ding disclose the limitations of claim 3. Ding also teaches wherein utilizing the similarity heuristic to determine similarity scores comprises utilizing a spatially constrained similarity measure with a voting map- based measuring approach (fig. 5; ¶44).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTOINETTE SPINKS whose telephone number is (571)270-3749. The examiner can normally be reached M-Th 7am - 5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at 571-272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTOINETTE T SPINKS/Primary Examiner, Art Unit 2639
Prosecution Insights