DETAILED ACTION
This office action is responsive to the response filed 7/17/2025. The application contains claims 1, 3-13, 15-20, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 1 objected to because of the following informalities: claim recites “executed to edit to the digital content” instead of “executed to edit the digital content”. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claim limitations in amended claim 12 have been interpreted under 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph, because it uses a non-structural term “edit recording system” coupled with functional language without reciting sufficient structure to achieve the function. Furthermore, the non-structural term is not preceded by a structural modifier.
Claim 12 recites the limitation "an edit recording system” coupled with functional language “generating” without reciting sufficient structure to achieve the function.
Since these claim limitations invoke 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph, claims 12-16 are interpreted to cover the corresponding structure described in the specification that achieves the claimed function, and equivalents thereof.
A review of the specification shows that the following appears to be the corresponding structure described in the specification for the 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph limitation:
Fig. 2, 116, 104, ¶27, “The computing device 102 is illustrated as including a content processing system 104. The content processing system 104 is implemented at least partially in hardware of the computing device 102 to process and transform digital content 106”, Based on the guidelines announced from Federal Register Vol. 76, No. 27, this has been interpreted as encompassing a hardware or hardware in combination with software implementation of the edit recording system, but not a pure software implementation.
If applicant wishes to provide further explanation or dispute the examiner’s interpretation of the corresponding structure, applicant must identify the corresponding structure with reference to the specification by page and line number, and to the drawing, if any, by reference characters in response to this Office action. Claimed modules also trigger interpretation of the claim language under 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph since they are considered a place holder for a corresponding structure in the specification.
If applicant does not wish to have the claim limitation treated under 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph, applicant may amend the claim so that it will clearly not invoke 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph, or present a sufficient showing that the claim recites sufficient structure, material, or acts for performing the claimed function to preclude application of 35 U.S.C. 112(f) or 35 U.S.C. 112 (pre-AIA ), sixth paragraph.
For more information, see MPEP § 2173 et seq. and Supplementary Examination Guidelines for Determining Compliance with 35 U.S.C. § 112 and for Treatment of Related Issues in Patent Applications, 76 FR 7162, 7167 (Feb. 9, 2011).
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 10 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 10 recites “automatically generating, the detecting, the generating, the receiving, the providing, the searching, the displaying execution of the at least one edit operation, and the displaying the logs are performed in real time as the user input is received via the user interface”. It is unclear how the user input that is used to identify a location within the display area that is logged in a log data for a search query could be done in real time with the creation of the log data associated with the editing operation that the same input is used to search within.
Claims 1-11 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites “logs identifying … edit operations performed at the indicated location”. It is unclear which indicated locations the claim limitations refers to as the claim disclose indicated locations outside and inside the rectangular area. For examination purposes examiner consider the indicated locations as the area within the selected area. Dependent claims inherit the independent claim deficiency.
Claims 12-16 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 12 recites “logs identifying … edit operations performed at the indicated location”. It is unclear which indicated locations the claim limitations refers to as the claim disclose indicated locations outside and inside the rectangular area. For examination purposes examiner consider the indicated locations as the area within the selected area. Dependent claims inherit the independent claim deficiency.
Claims 17-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 17 recites “logs identifying … edit operations performed at the indicated location”. It is unclear which indicated locations the claim limitations refers to as the claim disclose indicated locations outside and inside the rectangular area. For examination purposes examiner consider the indicated locations as the area within the selected area. Dependent claims inherit the independent claim deficiency.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-13, 15-16 are rejected under 35 U.S.C. § 101 because the instant application is directed to non-patentable subject matter. Specifically, the claims are directed toward at least one judicial exception without reciting additional elements that amount to significantly more than the judicial exception. The rationale for this determination is in accordance with the guidelines of USPTO, applies to all statutory categories, and is explained in detail below.
When considering subject matter eligibility under 35 U.S.C. 101, (1) it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, (2a) it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so (2b), it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself. Examples of abstract ideas include certain methods of organizing human activities; a mental processes; and mathematical concepts, (2019 PEG)
STEP 1.
Per Step 1, the claims fall within one of the statutory categories as in independent Claim 1, 12 in the therefrom dependent claims. Therefore, the claims are directed to a statutory eligibility category.
Step 2A: The invention is directed to searching a log data based on a received search request and providing the results to a user (Mental Process). As such, the claims include an abstract idea.
The limitation that has been identified as an abstract idea:
“generating, log data that describes a plurality of edit operations executed to edit digital content, the log data including location data indicating a plurality of locations within the digital content at which the plurality of edit operations are executed” (Mental Process, user can generate log data); “user input identifying a location within the digital content displayed in a user interface, the location defining an area within the content” (Mental Process, user can select a point within displayed content); “generating, a search query based on the location, the search query including a set of coordinates specifying a boundary of the area within the content” (Metal Process, user can start a search based on a point within displayed content), “searching and based on the search query the log data by filtering out logs of the log data associated with location data indicating locations outside of the area; and obtaining, based on the filtering, a search result including logs of the log data associated with location data indicating locations disposed within the area, the logs identifying the location data and at least one edit operation of the plurality of edit operations performed at the indicated locations” (Mental Process, user can identify logs associated with a specific location with a displayed content).
This judicial exception is not integrated into a practical application.
The claim recite additional elements as “method implemented by a computing device”, “by the computing device”, “digital content” are limitations that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f)(2), “detecting, by the computing device, a user input” (Insignificant Extra-Solution Activity, MPEP 2106.05(g)), “generating, by the computing device”, “providing, by the computing device” , “receiving, by the computing device”, “displaying, by the computing device”, are limitations that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f)(2), in addition does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)(1)). “TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea”).
The elements are recited at a high level of generality, i.e. a generic computing system performing generic functions including generic processing and receiving of data. Accordingly the additional elements do not integrate the abstract into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore the claims are directed to an abstract idea. (2019 Revised Patent Subject Matter Eligibility Guidance ("2019 PEG"). Thus, under Step 2A of the Mayo framework, the Examiner holds that the claims are directed to concepts identified as abstract.
STEP 2B.
Because the claims include one or more abstract ideas, the examiner now proceeds to Step 2B of the analysis, in which the examiner considers if the claims include individually or as an ordered combination limitations that are "significantly more" than the abstract idea itself. This includes analysis as to whether there is an improvement to either the "computer itself," "another technology," the "technical field," or significantly more than what is "well-understood, routine, or conventional" in the related arts.
The instant application includes in Claim 1 additional steps to those deemed to be abstract idea.
“method implemented by a computing device”, “by the computing device”, “digital content” are limitations that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f)(2), in addition does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it", “detecting, by the computing device, a user input” (Insignificant Extra-Solution Activity, MPEP 2106.05(g), and based on court decisions well understood, routine and conventional computer functions or mere instruction and/or insignificant activity have been identified to include: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321,120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); In Bilski referring to Flook, where Flook determined that an insignificant post-solution activity does not makes an otherwise patent ineligible claim patent eligible. In Bilski, the court added to Flook that pre-solution (such as data gathering) and insignificant step in the middle of a process (such as receiving user input) to be equally ineffective, MPEP 2106.05 (d)) that does not integrate a judicial exception into a practical application or provide significantly more, “device”, are limitations that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f)(2)) and does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)(1)). “TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea”). Examiner further notes that receiving data, user input and generating output is Well-Understood, Routine, Conventional Activity.
In the instant case, Claim 1 is directed to above mentioned abstract idea. Technical functions such as receiving, and processing data are common and basic functions in computer technology. The individual limitations are recited at a high level and do not provide any specific technology or techniques to perform the functions claimed.
Looking to MPEP 2106.05 (d), based on court decisions well understood, routine and conventional computer functions or mere instruction and/or insignificant activity have been identified to include: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321,120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); In Bilski referring to Flook, where Flook determined that an insignificant post-solution activity does not makes an otherwise patent ineligible claim patent eligible. In Bilski, the court added to Flook that pre-solution (such as data gathering) and insignificant step in the middle of a process (such as receiving user input) to be equally ineffective. The claims does not provide any specific process with respect to the additional elements that would transform the function beyond what is well understood. Like as found in Electric Power Group, Bilski, the technical process to implement the input and display functions are conventional and well understood.
In addition, when the claims are taken as a whole, as an ordered combination, the combination of steps does not add "significantly more" by virtue of considering the steps as a whole, as an ordered combination. The instant application, therefore, still appears only to implement the abstract idea to the particular technological environments using what is well-understood, routine, and conventional in the related arts. The steps are still a combination made to the abstract idea. The additional steps only add to those abstract ideas using well-understood and conventional functions, and the claims do not show improved ways of, for example, an unconventional non-routine functions for analyzing documents or receiving user input or generating output that could then be pointed to as being "significantly more" than the abstract ideas themselves. Moreover, Examiner was not able to identify any "unconventional" steps, which, when considered in the ordered combination with the other steps, could have transformed the nature of the abstract idea previously identified. The instant application, therefore, still appears to only implement the abstract ideas to the particular technological environments using what is well-understood, routine, and conventional in the related arts.
Further, note that the limitations, in the instant claims, are done by the generically recited computing devices. The limitations are merely instructions to implement the abstract idea on a computing device that is recited in an abstract level and require no more than a generic computing devices to perform generic functions.
CONCLUSION
It is therefore determined that the instant application not only represents an abstract idea identified as such based on criteria defined by the Courts and on USPTO examination guidelines, but also lacks the capability to bring about "Improvements to another technology or technical field" (Alice), bring about "Improvements to the functioning of the computer itself" (Alice), "Apply the judicial exception with, or by use of, a particular machine" (Bilski), "Effect a transformation or reduction of a particular article to a different state or thing" (Diehr), "Add a specific limitation other than what is well-understood, routine and conventional in the field" (Mayo), "Add unconventional steps that confine the claim to a particular useful application" (Mayo), or contain "Other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment" (Alice), transformed a traditionally subjective process performed by humans into a mathematically automated process executed on computers (McRO), or limitations directed to improvements in computer related technology, including claims directed to software (Enfish).
The dependent claims, when considered individually and as a whole, likewise do not provide “significantly more” than the abstract idea for similar reasons as the independent claim. For example claim 3 disclose “wherein the log data includes: operation data describing the plurality of edit operations used to edit the digital content through interaction with the user interface of a content processing system; and time data indicating a time at which the plurality of edit operations are executed, respectively (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that doesn't integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 4, disclose” wherein the user input identifies the location by specifying the boundary within the digital content or a digital object displayed within the digital content in the user interface” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 5 disclose “generating a digital video depicting a timelapse sequence of the at least one edit operation from the search result as being used to edit the digital content, and wherein the displaying execution of the at least one edit operation is performed using the digital video” (extra insignificant solution that is well-understood, routine, and conventional) as shown by Titi et al. [US 20150350591 A1] See at least ¶13, William et al. [US 20150350544 A1] See at least ¶¶5-6, Park et al. [US 20190364208 A1] See at least ¶75, and does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)(1)), claim 6, disclose “the digital video depicts selection of representations used to initiate the execution of the at least one edit operation” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 7 disclose “user input further identifies a particular edit operation of the plurality of edit operations, the search query specifies the particular edit operation, and the at least one edit operation of the search result corresponds to the particular edit operation as performed at the location” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 8 disclose “user input includes selecting a representation from a plurality of representations of edit operations used to edit the digital content, the plurality of representations generated by searching the log data associated with the digital content” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 9 disclose “user input further identifies a digital object within the digital content; the search query specifies the digital object; and the at least one edit operation in the search result corresponds to the digital object” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 10 disclose “automatically generating, the detecting, the generating, the receiving, the providing, the searching, the displaying execution of the at least one edit operation, and the displaying the logs are performed in real time as the user input is received via the user interface” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 11 disclose “wherein the digital content is a digital image” (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea.
The dependent claims which impose additional limitations also fail to claim patent-eligible subject matter because the limitations cannot be considered statutory. The dependent claim(s) have been examined individually and in combination with the preceding claims, however they do not cure the deficiencies of claim 1; where all claims are directed to the same abstract idea, "addressing each claim of the asserted patents [is] unnecessary." Content Extraction &. Transmission LLC v, Wells Fargo Bank, Natl Ass'n, 776 F.3d 1343, 1348 (Fed. Cir. 2014). If applicant believes the dependent claims are directed towards patent eligible subject matter, they are invited to point out the specific limitations in the claim that are directed towards patent eligible subject matter. Claims for the other statutory classes are similarly analyzed. For example claim 12 extra limitation of outputting a selectable list of data is extra insignificant solution that is well-understood, routine, and conventional) as shown by AVOYAN et al. [US 2021/0027510 A1] See at least Fig. 7, Fig. 3A, ¶69, ¶70, Havoc Pennington et al. [US 11,687,212 B2] See at least Fig. 4-6, Fig. 8 Quinn et al. [US 20100241507 A1] See at least Fig. 8, and does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)(1)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al. [US 2013/0120439 A1, hereinafter Harris] in view of Brouwer et al. [US 2019/0096439 A1, hereinafter Ahmed].
With regard to Claim 1,
Harris teach in a digital medium environment, a method implemented by a computing device, the method comprising:
automatically generating, by the computing device, log data that describes a plurality of edit operations executed to edit digital content, the log data including location data indicating a plurality of locations within the digital content at which the plurality of edit operations are executed (Fig. 1, Fig. 7, ¶¶3-4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶¶5-6, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶105, ¶113, ¶¶35-36, “frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps. In other embodiments, if the frame identifier of each frame includes a frame number, each entry of the log may include a corresponding frame number, rather than to a timestamp, and these frame numbers may be matched up to determine a correlation between frames of the animation and the log entries”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”, specific frame and portions of image is a location data within digital content);
detecting, by the computing device, a user input identifying a location within the digital content displayed in a user interface (Abstract, ¶42, “if the user wishes to undo an image editing operation, they may select a visual rewind operation in the image editing application”, ¶95, “image editing application may provide mechanisms to allow the user to navigate to the point in the animation displayed by the visual rewind operation corresponding to the image state to which the user wants to return”, ¶¶4-5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier”, user identify a location (time point/frame) within the media displayed in a user interface);
generating, by the computing device, a search query based on the location, the search query identifying the location (Fig. 1, Fig. 7, ¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶10, ¶¶35-36, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, navigating to a specific frame (location) is a search query);
providing, by the computing device, the search query as input to a search module (¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶¶42-45, navigating to a specific timestamp of the media (location) is a search query);
searching, by the search module and based on the search query, the log data by filtering out logs of the log data associated with location data indicating locations outside of the [identified location] (¶4, “frame identifier may include a frame number and/or a timestamp, ¶5, “each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”, identifying frames based on the timestamp selected by the user rewinding to a specific time point will exclude any of the frames not associated with the selected frame); and
obtaining, based on the filtering, a search result including logs of the log data associated with location data indicating locations, the logs identifying the location data and at least one edit operation of the plurality of edit operations performed at the indicated locations (¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶34, “As a log generator or generates and stores entries in the interaction log, a timestamp and/or frame number may be recorded in the each log entry“, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”);
receiving, by the computing device, the search result, displaying, by the computing device, execution of the at least one edit operation on the digital content in the user interface based on the search result (Fig. 1, Fig. 7, ¶6, “system may be configured to store the captured image data, the log data, and data representing the correlation between each of the plurality of entries and the respective frame identifier. In performing a subsequent operation of the image editing application (e.g., a visual rewind operation)”, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”); and
displaying, by the computing device, logs identifying the location data (¶4, “The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”) and identifying the at least one edit operation of the plurality of edit operations performed at the indicated locations (¶4, “The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”), the logs searchable to locate a respective said edit operation based on a respective said location at which the respective said edit operation was executed to edit to the digital content (¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶4, “The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”).
Harris does not explicitly teach the location defining an area within the digital content; search query including set of coordinates specifying a boundary of the area; location data indicating locations disposed within the area.
Ahmed teach in a digital medium environment, a method implemented by a computing device, the method comprising:
automatically generating, by the computing device, log data that describes a plurality of edit operations executed to edit digital content, the log data including location data indicating a plurality of locations within the digital content at which the plurality of edit operations are executed (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”);
detecting, by the computing device, a user input identifying a location within the digital content displayed in a user interface, the location defining an area within the digital content (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶53, “During playback the system will display the Screen Tag for a duration of, for example, 4 seconds and the user may in this time have sufficient time to click and explore the Screen Tag”);
generating, by the computing device, a search query based on the location, the search query including a set of coordinates specifying a boundary of the area within the digital content (Fig. 12, ¶16, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶64, “object tracking is used differently. In the present invention, object tracking is used to detect and track a collection of pixels from a location selected by a user corresponding to an object for a predetermined time duration”, ¶¶67-68, “a period long enough for a user to see a Screen Tag and to interact with it when a video is running showing at least one Screen Tag”, ¶76, “The idea is to offer a method of grouping Screen Tags shown on screen so that they can be displayed, filtered, searched or hidden by the viewer”, ¶88, “Screen Tags are used, the clickable Screen Tags (20) will follow their positively identified elements … and will then disappear unless a user decides to click on them, which will cause the video to pause to reveal the information of the tag”);
Providing, by the computing device, the search query as input to a search module ¶¶58-59, “system may use one of several known methods for tracking the collection of pixels around the location where the user wants to insert the tag”; searching by the searching module and based on the search query, the log data by:
Filtering out logs of the log data associated with location data indicating locations outside of the area (¶50, “Screen Tag can be clicked to reveal the associated type of content”, ¶¶58-59, “system may use one of several known methods for tracking the collection of pixels around the location where the user wants to insert the tag”, ¶63, “pixel tracking analysis is only required for identifying a specific element the user clicked on“); and
obtaining, based on the filtering, a search result including logs of the log data associated with location data indicating locations disposed within the area, the logs identifying the location data and at least one edit operation of the plurality of edit operations performed at the indicated locations (Fig. 12, Fig. 24, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 302, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶52. “ Pixel region tracking is used to determine whether the collection of pixels in proximity to the location identified by the user can be tracked for a predetermined length of, for example, 4 seconds. The system will determine whether the pixels are trackable over this period so that a Screen Tag can be placed at the identified position“, adding a tag to a specific location is editing the displayed media content and selecting a tag that is associated with the annotation and displayed in the annotation location is a search for the location associated data (excluding other locations) that could include attached videos);
receiving, by the computing device, the search result (Fig. 12, Fig. 24, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 302, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶¶58-60, “object tracking analysis to determine whether or not the pixels corresponding to the selected object at the clicked location”, adding a tag to a specific location is editing the displayed media content and selecting a tag that is associated with the annotation and displayed in the annotation location is a search for the location associated data that could include attached videos), and
displaying, by the computing device, [video] on the digital content in the user interface based on the search result (Fig. 10, 40, ¶¶78-79, “as shown in FIG. 10, a user might add a description (37) or add, or link a file, or a video (40) or other data to the Description Tag”, Fig. 12, Fig. 24, 300, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”); and
displaying, by the computing device, the logs identifying the location data and identifying the at least one edit operation of the plurality of edit operations performed at the indicated locations,, the log data searchable to locate a respective said edit operation based on a respective said location at which the respective said edit operation was executed to edit to the digital content (Fig. 10, 40, ¶¶78-79, “as shown in FIG. 10, a user might add a description (37) or add, or link a file, or a video (40) or other data to the Description Tag”, Fig. 12, Fig. 24, 300-317, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”, represented tags are displayed data that could be searched to locate a respective edit operation by clicking on it to display the edit operation (associated annotation or video)).
In other words, Harris teach the ability to store, retrieve and display editing process for digital content. Harris further teach the ability to create a region list that represent the regions of the images that have been changed and the ability to animate and display the edit operation corresponding to any of the regions (Fig, 4, ¶ “CPU function 424, configured to build an optimal region list from the bit field, representing the regions of the image that have changed since the previous frame. A buffer 430 used by the GPU and CPU to pack and unpack the region of interest.”, ¶83, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs. For example, by storing information indicating the portion of the image in which changes are made, the system may support a highlighting feature when the animation is replayed”, ¶105).Harris does not explicitly teach the ability to search based on user’s selected region coordinates within displayed content as Harris provide temporal search based on timestamps associated with frames . However, Ahmed teach the ability to display different tags that could be displayed on edited areas (adding comment or annotation or associating a file is editing) and are searchable as user can select to create a search query that return the data associated with the searched location (Fig. 12).
Harris and Ahmed are analogous art to the claimed invention because they are from a similar field of endeavor of interacting with displayed content to identify and search for related data. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris to include the ability to use selected location coordinates and to display searchable log data to identify related editing data resulting in resolutions as disclosed by Ahmed with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data that can be easily selected and for user’s previous editing operation based on the users’ interest to save the users’ time and effort. This is simply combining prior art elements according to known methods to yield predictable results, usage of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 3,
Harris-Ahmed teach the method as described in claim 1, wherein the log data includes:
operation data describing the plurality of edit operations used to edit the digital content through interaction with the user interface of a content processing system (Harris, Fig. 1, ¶4, “capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶¶7-8, ¶34, “while image data for the animation is being captured, the method may include simultaneously recording a log of interactions and corresponding state changes (e.g., based on input received from the image editing application), which may then be correlated with frame numbers on the animation”); and
time data indicating a time at which the plurality of edit operations are executed, respectively (Harris, Fig. 6a, ¶92, “image editing application may also be configured to display a timeline 625, indicating (by a bold mark) which of a sequence of frames currently captured is being displayed, as identified by a timestamp and/or frame number along timeline 625”, ¶10, ¶¶35-36, “frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 4,
Harris-Ahmed teach the method as described in claim 1, wherein the user input identifies the location by specifying the boundary within the digital content or a digital object displayed within the digital content in the user interface (Harris, ¶105, Ahmed, Fig. 12, Fig. 24, 300, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 5,
Harris-Ahmed teach the method as described in claim 1, further comprising generating a digital video depicting a timelapse sequence of the at least one edit operation from the search result as being used to edit the digital content and wherein the displaying execution of the at least one edit operation is performed using the digital video (Harris, ¶43, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation. In other words, the system may be configured to allow a user to arbitrarily (and visually) navigate an undo/redo queue of the image editing application”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 9,
Harris-Ahmed teach the method as described in claim 1, wherein:
the user input further identifies a digital object within the digital content; the search query specifies the digital object; and the at least one edit operation in the search result corresponds to the digital object (Harris, Fig. 6A-6D, “select object”, ¶105, “user may be prompted to select an object or a portion of the input image 605 being edited in active editing window 610 on which to apply the operation”, “if an image editing operation is applied to a selected object or portion of an image, the image editing application may provide data indicating the selection to the interaction log generation process as a hint about the region of interest in the corresponding animation frames”, Ahmed, Fig. 12, Fig. 24, 300, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 10,
Harris-Ahmed teach the method as described in claim 1, wherein the automatically generating, the detecting, the generating, the receiving, the providing, the searching, the displaying execution of the at least one edit operation, and the displaying the logs are performed in real time as the user input is received via the user interface (Harris, (¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶4, “The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, Ahmed, Fig. 12, Fig. 24, 300, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 11,
Harris-Ahmed teach the method as described in claim 1, wherein the digital content is a digital image (Harris, Fig. 6A-6D, Ahmed, ¶43, “present invention relates to a method, process and system for contextually augmenting and annotating moving pictures or images with tags using pixel region tracking on computing devices with screen display”, ¶49, “Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image”).
The same motivation to combine for claim 1 equally applies for current claim.
Claims 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al. [US 2013/0120439 A1, hereinafter Harris] in view of Brouwer et al. [US 2019/0096439 A1, hereinafter Ahmed] further in view of AVOYAN et al. [US 2021/0027510 A1, hereinafter AVOYAN].
With regard to Claim 6,
Harris-Ahmed teach the method as described in claim 5, wherein digital video (Harris, ¶43, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”). The same motivation to combine for claim 5 equally applies for current claim.
Harris-Ahmed do not explicitly teach depicts selection of representations used to initiate execution of the at least one edit operations.
AVOYAN teach digital video depicts selection of representations used to initiate execution of the at least one edit operations (Fig. 4, 414, 416, 418a, 418n, Fig. 10A-15, ¶85, “player application reads the replay file with the edit instructions and related resources and controls the edit tool opening sequence and actions. For example, in embodiments, the player reads the JSON and determines the exact crop size/position parameter so the user may simply accept and apply that or adjust which part/size/position of their image they wish to crop. As shown, the display has an indicator 1015 that indicates which edit tool is currently active (e.g., “Crop”) and may indicate which step in the replay sequence it is (e.g., Step 1 of 4, in this example)”, ¶86).
Harris-Ahmed and AVOYAN are analogous art to the claimed invention because they are from a similar field of endeavor of editing images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris-Ahmed to include the ability to depicts selection of representations used to initiate execution of the edit operations resulting in resolutions as disclosed by AVOYAN with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris-Ahmed as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data based on the users’ interest to save the users’ time and effort allowing content creators to easily and efficiently share the information of their editing processes with others and systems and methods that allow others to apply the same or similar edits in a simple and efficient manner (AVOYAN, ¶7).
With regard to Claim 7,
Harris-Ahmed teach the method as described in claim 1.
The same motivation to combine for claim 1 equally applies for current claim.
Harris-Ahmed do not explicitly teach wherein the user input further identifies a particular edit operation of the edit operations, the search query specifies the particular edit operation, and the edit operations of the search result correspond to the particular edit operation as performed at the location.
AVOYAN teach wherein the user input further identifies a particular edit operation of the at least one edit operations, the search query specifies the particular edit operation, and the plurality of edit operations of the search result correspond to the particular edit operation as performed at the location (Fig. 3A, 306b-306m, ¶69, “user's feed may display a replay card 302 that includes an image display section 304; a first thumbnail 306 a that shows the original image; and other thumbnails 306b-306m representing the sequence of editing tools/steps, where the thumbnails may correspond to the tools or steps that the creator used to edit the original image”, ¶70, “user may pause over or select the edit card (e.g., 306d, 306m, etc.) and see that result of that specific edit”).
Harris-Ahmed and AVOYAN are analogous art to the claimed invention because they are from a similar field of endeavor of editing images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris-Ahmed to include the ability to depicts selection of representations used to initiate execution of the edit operations resulting in resolutions as disclosed by AVOYAN with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data based on the users’ interest to save the users’ time and effort allowing content creators to easily and efficiently share the information of their editing processes with others and systems and methods that allow others to apply the same or similar edits in a simple and efficient manner (AVOYAN, ¶7).
With regard to Claim 8,
Harris-Ahmed-AVOYAN teach the method as described in claim 7, wherein the user input includes selecting a representation from a plurality of representations of edit operations used to edit the digital content, the plurality of representations generated by searching the log data associated with the digital content (Harris, Fig. 1, Fig. 7, ¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation)”, AVOYAN, Fig. 3A, 306b-306m, ¶69, “depicted, the user's feed may display a replay card 302 that includes an image display section 304; a first thumbnail 306 a that shows the original image; and other thumbnails 306 b-306 m representing the sequence of editing tools/steps, where the thumbnails may correspond to the tools or steps that the creator used to edit the original image”, ¶70, “user may pause over or select the edit card (e.g., 306 d, 306 m, etc.) and see that result of that specific edit”).
The same motivation to combine for claim 7 equally applies for current claim.
Claims 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al. [US 2013/0120439 A1, hereinafter Harris] in view of Brouwer et al. [US 2019/0096439 A1, hereinafter Ahmed] in view of Suzuki et al. [US 20090055094 A1, hereinafter Suzuki] further in view of AVOYAN et al. [US 2021/0027510 A1, hereinafter AVOYAN].
With regard to Claim 12,
Harris teach a computing device comprising:
a processing device; and a computer-readable storage medium storing instructions that, responsive to execution by the processing device, causes the processing device to perform operations including (Harris, ¶7):
receiving user inputs specifying a plurality of edit operations to edit digital content displayed in a user interface (Harris, Fig. 6a-6D, selectable icons associated with editing options);
editing the digital content using the specified plurality of edit operations (Harris, Fig. 6a-6D, Fig. 7, Fig. 8);
automatically generating, using an edit recording system, log data describing the plurality of edit operations (Harris, Fig. 1, ¶4, “capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations” ¶¶7-8, ¶34), the log data including location data indicating a plurality of locations at which the plurality of edit operations are executed within the digital content (Harris, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs. For example, by storing information indicating the portion of the image in which changes are made” Fig. 1, Fig. 7, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶105, ¶113, ¶¶35-36, “frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps. In other embodiments, if the frame identifier of each frame includes a frame number, each entry of the log may include a corresponding frame number, rather than to a timestamp, and these frame numbers may be matched up to determine a correlation between frames of the animation and the log entries”, specific frame and portions of image is a location data within digital content),
outputting the digital content as edited using the plurality of edit operations and the log data as associated with the digital content, the log data searchable to locate a respective said edit operation, at which, the respective said edit operation was executed to edit to the digital content (¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, Fig. 1, Fig. 7, ¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶10, ¶¶35-36, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, navigating to a specific frame (location) is a search query);
receiving a user input identifying a location within the edited digital content (¶10, ¶¶35-36, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, navigating to a specific frame (location) is a user input identifying a location with edited digital content);
generating a search query based on the location, the search query including a set of coordinates specifying [location] of the digital content (Fig. 1, Fig. 7, ¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶10, ¶¶35-36, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, navigating to a specific frame (location) is a search query);
generating a search result based on the search query by:
filtering out logs of the log data associated with location data indicating locations outside of the [selected location] (¶4, “frame identifier may include a frame number and/or a timestamp, ¶5, “each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”, identifying frames based on the timestamp selected by the user rewinding to a specific time point and exclude any of the frames not associated with the selected frame); and
obtaining, based on the filtering, the search result including logs of the log data associated with location data indicating locations disposed within the [selected location], the logs identifying the location data and at least one edit operation of the plurality of edit operations performed at the indicated locations (¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶34, “As a log generator or generates and stores entries in the interaction log, a timestamp and/or frame number may be recorded in the each log entry“, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”); and
by searching the log data using the location, the search result including representations of log data corresponding to edit operations performed at the location to edit the digital content (Fig. 1, Fig. 7, ¶6, “system may be configured to store the captured image data, the log data, and data representing the correlation between each of the plurality of entries and the respective frame identifier. In performing a subsequent operation of the image editing application (e.g., a visual rewind operation)”, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”)
outputting the search result of representations of the logs identifying the location data (¶4, “The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”) and the at least one edit operation of the plurality of edit operations performed at the indicated locations (¶4, “The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”), the representations of log data the logs selectable to navigate to corresponding configurations of the digital content as being edited by the at least one edit operation (¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”)
Harris does not explicitly teach location defining an area within the digital content; a set of coordinates specifying a boundary of the area of the digital area; outside of the area, within the area.
Ahmed teach data automatically generating, using an edit recording system, log data describing the plurality of edit operations, the log data including location data indicating a plurality of locations at which the plurality of edit operations are executed within the digital content (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position (12) and the frame number”, ¶50, tags associated with specific coordinates that identify a display location within specific frames)
outputting the digital content as edited using the plurality of edit operations and the log data as associated with the digital content, the log data searchable to locate a respective said edit operation, at which, the respective said edit operation was executed to edit to the digital content (Fig. 2, 13, Fig. 6-11, 13, ¶46, “computing device (14) features standard controls known to video streaming applications”, ¶56, “a play button, or other video control functions or buttons, are visible or are becoming visible (13) when a user clicks or interacts with the screen as shown in FIG. 6”, ¶49, “anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position (12) and the frame number”, ¶50, Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, tags associated with specific locations within specific frames and user search by clicking on the specific coordinates within the display or by moving to specific frame that the tag is associated with using video control features),
outputting the digital content as edited using the plurality of edit operations and the log data as associated with the digital content, the log data searchable to locate a respective said edit operation, at which, the respective said edit operation was executed to edit to the digital content (Fig. 12, ¶16, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶64, “object tracking is used differently. In the present invention, object tracking is used to detect and track a collection of pixels from a location selected by a user corresponding to an object for a predetermined time duration”, ¶¶67-68, “a period long enough for a user to see a Screen Tag and to interact with it when a video is running showing at least one Screen Tag”, ¶76, “The idea is to offer a method of grouping Screen Tags shown on screen so that they can be displayed, filtered, searched or hidden by the viewer”, ¶88, “Screen Tags are used, the clickable Screen Tags (20) will follow their positively identified elements … and will then disappear unless a user decides to click on them, which will cause the video to pause to reveal the information of the tag”, Fig. 10-11, Fig. 24, 300-317, ¶53, “During playback the system will display the Screen Tag for a duration of, for example, 4 seconds and the user may in this time have sufficient time to click and explore the Screen Tag”, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”, tags that represent edited digital content include searchable log data that associate them with specific frames and locations or coordinates within the frames);
receiving a user input identifying a location within the edited digital content, the location defining an area within the digital content (Fig. 12, ¶16, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶64, “object tracking is used differently. In the present invention, object tracking is used to detect and track a collection of pixels from a location selected by a user corresponding to an object for a predetermined time duration”, ¶¶67-68, “a period long enough for a user to see a Screen Tag and to interact with it when a video is running showing at least one Screen Tag”, ¶76, “The idea is to offer a method of grouping Screen Tags shown on screen so that they can be displayed, filtered, searched or hidden by the viewer”, ¶88, “Screen Tags are used, the clickable Screen Tags (20) will follow their positively identified elements … and will then disappear unless a user decides to click on them, which will cause the video to pause to reveal the information of the tag” Fig. 2, 13, Fig. 6-11, 13, ¶46, “computing device (14) features standard controls known to video streaming applications”, ¶56, “a play button, or other video control functions or buttons, are visible or are becoming visible (13) when a user clicks or interacts with the screen as shown in FIG. 6”, ¶49, “anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position (12) and the frame number”, ¶50, tags associated with specific locations within specific frames and user search by navigating to specific frame (location) using video control features that the tag is associated with);
generating a search query based on the location, the search query including a set of coordinates specifying a boundary of the area of the digital content (Fig. 12, ¶16, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶64, “object tracking is used differently. In the present invention, object tracking is used to detect and track a collection of pixels from a location selected by a user corresponding to an object for a predetermined time duration”, ¶¶67-68, “a period long enough for a user to see a Screen Tag and to interact with it when a video is running showing at least one Screen Tag”, ¶76, “The idea is to offer a method of grouping Screen Tags shown on screen so that they can be displayed, filtered, searched or hidden by the viewer”, ¶88, “Screen Tags are used, the clickable Screen Tags (20) will follow their positively identified elements … and will then disappear unless a user decides to click on them, which will cause the video to pause to reveal the information of the tag”);
generating a search result by:
Filtering out logs of the log data associated with location data indicating locations outside of the area (¶50, “Screen Tag can be clicked to reveal the associated type of content”, ¶¶58-59, “system may use one of several known methods for tracking the collection of pixels around the location where the user wants to insert the tag”, ¶63, “pixel tracking analysis is only required for identifying a specific element the user clicked on“); and
obtaining, based on the filtering, the search result including logs of the log data associated with location data indicating locations disposed within the area, the logs identifying the location data and at least one edit operation of the plurality of edit operations performed at the indicated locations (Fig. 12, Fig. 24, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 302, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, adding a tag to a specific location is editing the displayed media content and selecting a tag that is associated with the annotation and displayed in the annotation location is a search for the location associated data (excluding other locations) that could include attached videos);
outputting the search result of representations of log data, the representations of the logs identifying the location data and the at least one edit operation of the plurality of edit operations performed at the indicated locations, the representation of the logs selectable to navigate to corresponding configurations of the digital content as being edited by the at least one edit operations (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 302, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, adding a tag to a specific location is editing the displayed media content and selecting a tag that is associated with the annotation and displayed in the annotation location is a search for the location associated data that could include attached videos, Fig. 2, 13, Fig. 6-11, 13, ¶46, “computing device (14) features standard controls known to video streaming applications”, ¶56, “a play button, or other video control functions or buttons, are visible or are becoming visible (13) when a user clicks or interacts with the screen as shown in FIG. 6”, ¶49, the displayed tags associated with the selected frame is a search result).
Harris and Ahmed are analogous art to the claimed invention because they are from a similar field of endeavor of interacting with displayed content to identify and search for related data. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris to include the ability to use selected location coordinates and to display searchable log data to identify related editing data resulting in resolutions as disclosed by Ahmed with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data that can be easily selected and for user’s previous editing operation based on the users’ interest to save the users’ time and effort. This is simply combining prior art elements according to known methods to yield predictable results, usage of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Harris-Ahmed does not explicitly disclose that the location defining a rectangular area.
Suzuki teach receiving a user input identifying a location within the edited digital content, the location defining an area within the digital content (Fig. 6, ¶54, “as shown in FIG. 6, the CPU 20 of the PND 1 overlaps and displays a rectangle search subject area setting frame SA1 which is slightly larger than the fingertip of the user on the navigation map image”, “search subject area setting frame SA1 touched by the fingertip of the user is traced in an arbitrary direction desired by the user, the CPU 20 of the PND 1 enlarges the rectangle size in this direction such that the diagonal of the search subject area setting frame SA1 is extended, or reduces the rectangle size in this direction such that the diagonal of the search subject area setting frame SA1 is reduced, and the rectangle size is fixed when the fingertip of the user is released from the screen”).
Harris-Ahmed and Suzuki ed are analogous art to the claimed invention because they are from a similar field of endeavor of interacting with displayed content to identify and search for related data. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris-Ahmed resulting in resolutions as disclosed by Suzuki with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris-Ahmed as described above to provide users with a simple and fast form of searching that include only the registered point information of an area which is desired by a user which will increase the users satisfaction by providing results that meets the users’ interest to save the users’ time and effort (Suzuki, ¶6).
Even though Ahmed appears to display search results as lists (¶82, “Tag Containers may in another embodiment be shown in a list view in a separate window”).
However, in effort to expedite prosecution AVOYAN explicitly teach generating a search result based on a search query (Fig. 5-6, ¶75, “feed 500 contains several images, which may be from different third parties. In the depicted embodiment, the feed contains at least one remix image 505, which is denoted by a replay indicator 510 to indicate that there is associated replay edit data for duplicating the edits. In one or more embodiments, a profile feed page may be one type of feed page; other types of feed pages may include pages comprising content from users that a user follows”, ¶76, user select an image that trigger the system to search the replay edit data associated with selected image for replay),
outputting the search result as a list of representations of the logs identifying at least one edit operation of the plurality of edit operations performed at an indicated location, the representations of logs selectable to navigate to corresponding configurations of the digital content as being edited by the at least one edit operation (Fig. 7A, 7B, 705, 715, Fig. 3A, ¶69, “first thumbnail 306 a that shows the original image; and other thumbnails 306 b-306 m representing the sequence of editing tools/steps, where the thumbnails may correspond to the tools or steps that the creator used to edit the original image”, ¶70, “user may pause over remix image for a certain period of time, and then, the steps in the replay may auto-play images on the image display section 304 so that a user may observe the sequential changes to the image at each editing step to that point. Thus, in one or more embodiments, the replay edit file may include a sequence of images depicting a progression of the original image to the edited image as the sequence of edits are applied, which may be stepped through or played like a video. In one or more embodiments, during the auto-play, a thumbnail may be highlighted to indicate which step (tool) is being taken. Alternatively, or additionally, the user may pause over or select the edit card (e.g., 306 d, 306 m, etc.) and see that result of that specific edit”).
Harris-Ahmed-Suzuki and AVOYAN are analogous art to the claimed invention because they are from a similar field of endeavor of editing images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris-Ahmed-Suzuki to include the ability to display the editing search results as a list of selectable elements resulting in resolutions as disclosed by AVOYAN with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris-Ahmed-Suzuki as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data based on the users’ interest to save the users’ time and effort allowing content creators to easily and efficiently share the information of their editing processes with others and systems and methods that allow others to apply the same or similar edits in a simple and efficient manner (AVOYAN, ¶7).
With regard to Claim 13,
Harris-Ahmed-Suzuki-AVOYAN teach the computing device as described in claim 12, wherein the log data is searchable to generate an additional search result as a digital video depicting a timelapse sequence of the at least one edit operations from the search result as being used to edit the digital content (Harris, ¶43, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation. In other words, the system may be configured to allow a user to arbitrarily (and visually) navigate an undo/redo queue of the image editing application”).
The same motivation to combine for claim 12 equally applies for current claim.
With regard to Claim 15,
Harris-Ahmed-Suzuki-AVOYAN teach the computing device as described in claim 13, the operations further comprising generating a digital video depicting a timelapse sequence of the plurality of edit operations based on the search result (Harris, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”).
The same motivation to combine for claim 12 equally applies for current claim.
With regard to Claim 16,
Harris-Ahmed-Suzuki-AVOYAN teach the computing device as described in claim 15, the operation further comprising posting the digital video to a content sharing service for access by client devices via a network (AVOYAN Fig. 16A-16C, ¶62, ¶96, “video of the replay may be separately shared via one or more networked services. For example, the video may be shared on as an Instagram story”, ¶97, “user may the QR code via in-app native QR scanners, a third-party app, or QR scanner that is part of the image editing application (e.g., PicsArt app)”).
Harris-Ahmed-Suzuki and AVOYAN are analogous art to the claimed invention because they are from a similar field of endeavor of editing images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris-Ahmed-Suzuki to include the ability to post editing videos to sharing services resulting in resolutions as disclosed by AVOYAN with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris-Ahmed-Suzuki as described above to share with other users or students or friends clear instructions regarding every editing step and how it is done to allow users to replicate and learn from the video as an educational tutorial, which will add more value, benefits and more usage for the captured editing steps video (AVOYAN, ¶7).
Claims 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Harris et al. [US 2013/0120439 A1, hereinafter Harris] in view of Brouwer et al. [US 2019/0096439 A1, hereinafter Ahmed] in view of Bradley et al. [US 2018/0090175 A1, hereinafter Bradley].
With regard to Claim 17,
Harris teach in a digital medium environment, a non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising (Harris, claim 12, ¶107);
executing a plurality of edit operations to edit digital content displayed in a user interface (Harris, Fig. 6a-6D, selectable icons associated with editing options);
automatically generating log data that describes the plurality of edit operations, the log data including location data indicating a plurality of locations within the digital content at which the plurality of edit operations are executed and time data describing respective times at which the plurality of edit operations are executed (Fig. 1, Fig. 7, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶105, ¶113, ¶¶35-36, “frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”, specific frames that identify an editing process and portions of image that are associated with the editing process are location data within digital content);
generating a first digital video depicting a timelapse sequence of the plurality of edit operations as being used to edit the digital content (¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation. In other words, the system may be configured to allow a user to arbitrarily (and visually) navigate an undo/redo queue of the image editing application”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”);
detecting a user input identifying a location within the digital content (Abstract, ¶42, “if the user wishes to undo an image editing operation, they may select a visual rewind operation in the image editing application”, ¶95, “image editing application may provide mechanisms to allow the user to navigate to the point in the animation displayed by the visual rewind operation corresponding to the image state to which the user wants to return”, ¶¶4-5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier”, user identify a location (time point/frame) within the media displayed in a user interface),
generating a search query based on the location, the search query including a [location] of the digital content (Fig. 1, Fig. 7, ¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶10, ¶¶35-36, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, navigating to a specific frame (location) is a search query);
searching the log data based on the search query by filtering out logs of the log data associated with location data indicating locations outside of the area (¶4, “frame identifier may include a frame number and/or a timestamp, ¶5, “each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”, identifying frames based on the timestamp selected by the user rewinding to a specific time point and exclude any of the frames not associated with the selected frame); and
obtaining, based on the filtering, a set of logs of the log data associated with location data indicating locations, the logs identifying the location data and at least one edit operation of the plurality of edit operations performed at the indicated locations (¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶34, “As a log generator or generates and stores entries in the interaction log, a timestamp and/or frame number may be recorded in the each log entry“, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”);
generating a search result associated with the first digital video based on the set of logs, the search result including one or more portions of the first digital video that depict the at least one edit operation of the plurality of edit operations executed at the indicated locations (Fig. 1, Fig. 7, ¶98, “system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs. For example, by storing information indicating the portion of the image in which changes are made, the system may support a highlighting feature when the animation is replayed”, ¶3, “image editing application may be configured to capture, compress, and store image data and interaction logs, and to use the stored information in a visual rewind operation, in which a sequence of frames (e.g., an animation) depicting changes in an image as image editing operations are performed is displayed in reverse order”, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶10, ¶¶35-36, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, navigating to a specific frame is searching for a location within the editing process of the media content is a search query for an edit location); and
automatically generating a second digital video based on the search result by:
automatically extracting a plurality of video frames from the first digital video, the plurality of video frames corresponding to the one or more portions of the first digital video (¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation. In other words, the system may be configured to allow a user to arbitrarily (and visually) navigate an undo/redo queue of the image editing application”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”); and
wherein the second digital video depicts a timelapse sequence of the edit operations as having occurred at the location (¶25, “most users understand how to use this skill to navigate through a video to identify a desired frame or state. The system and methods described herein may provide a mechanism by which a user may navigate through an animation of the performance of one of or more image editing operations to identify a point at which an image state or document is correct or a point at which the user wishes to save and/or further process the depicted image”).
Harris does not explicitly teach the location defining an area within the digital content; search query including set of coordinates specifying a boundary of the area.
Ahmed teach in a digital medium environment, a non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
executing a plurality of edit operations to edit digital content displayed in a user interface (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶50);
automatically generating log data that describes the plurality of edit operations, the log data including location data indicating a plurality of locations within the digital content at which the plurality of edit operations are executed (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”);
detecting a user input identifying a location within the digital content, the location defining an area of the digital content (Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶53, “During playback the system will display the Screen Tag for a duration of, for example, 4 seconds and the user may in this time have sufficient time to click and explore the Screen Tag”);
generating a search query based on the location, the search query including a set of coordinates specifying a boundary of the area of the digital content (Fig. 12, ¶16, ¶50, “Screen Tag is an icon on screen depicting the type of content associated with that identified object. The Screen Tag can be clicked to reveal the associated type of content”, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, ¶64, “object tracking is used differently. In the present invention, object tracking is used to detect and track a collection of pixels from a location selected by a user corresponding to an object for a predetermined time duration”, ¶¶67-68, “a period long enough for a user to see a Screen Tag and to interact with it when a video is running showing at least one Screen Tag”, ¶76, “The idea is to offer a method of grouping Screen Tags shown on screen so that they can be displayed, filtered, searched or hidden by the viewer”, ¶88, “Screen Tags are used, the clickable Screen Tags (20) will follow their positively identified elements … and will then disappear unless a user decides to click on them, which will cause the video to pause to reveal the information of the tag”);
searching the log data based on the search query by:
Filtering out logs of the log data associated with location data indicating locations outside of the area (¶50, “Screen Tag can be clicked to reveal the associated type of content”, ¶¶58-59, “system may use one of several known methods for tracking the collection of pixels around the location where the user wants to insert the tag”, ¶63, “pixel tracking analysis is only required for identifying a specific element the user clicked on“); and
obtaining, based on the filtering, a set of logs of the log data associated with location data indicating locations disposed within the area, the logs identifying the location data and at least one edit operation of the plurality of edit operations executed at the indicated locations (Fig. 12, Fig. 24, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 302, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, adding a tag to a specific location is editing the displayed media content and selecting a tag that is associated with the annotation and displayed in the annotation location is a search for the location associated data (excluding other locations) that could include attached videos);
generating a search result associated with the first digital video based on the location set of logs, the search result including one or more portions of the first digital video that depict the at least one edit operation of the plurality of edit operations executed at the location indicated locations (Fig. 10, 40, ¶¶78-79, “as shown in FIG. 10, a user might add a description (37) or add, or link a file, or a video (40) or other data to the Description Tag”, Fig. 12, Fig. 24, 300-317, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”, represented tags are displayed data that could be searched to locate a respective edit operation by clicking on it to display the edit operation (associated annotation or video))
In other words, Harris teach the ability to store, retrieve and display editing process for digital content. Harris further teach the ability to create a region list that represent the regions of the images that have been changed and the ability to animate and display the edit operation corresponding to any of the regions (Fig, 4, ¶ “CPU function 424, configured to build an optimal region list from the bit field, representing the regions of the image that have changed since the previous frame. A buffer 430 used by the GPU and CPU to pack and unpack the region of interest.”, ¶83, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs. For example, by storing information indicating the portion of the image in which changes are made, the system may support a highlighting feature when the animation is replayed”, ¶105).Harris does not explicitly teach the ability to search based on user’s selected region coordinates within displayed content as Harris provide temporal search based on timestamps associated with frames . However, Ahmed teach the ability to display different tags that could be displayed on edited areas (adding comment or annotation or associating a file is editing) and are searchable as user can select to create a search query that return the data associated with the searched location (Fig. 12).
Harris and Ahmed are analogous art to the claimed invention because they are from a similar field of endeavor of interacting with displayed content to identify and search for related data. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris to include the ability to use selected location coordinates and to display searchable log data to identify related editing data resulting in resolutions as disclosed by Ahmed with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data that can be easily selected and for user’s previous editing operation based on the users’ interest to save the users’ time and effort. This is simply combining prior art elements according to known methods to yield predictable results, usage of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Harris-Ahmed does not explicitly teach automatically smoothing a transition between each video frame of the plurality of video frames.
Bradley teach automatically generating a second digital video by: automatically extracting a plurality of video frames from the first digital video, the plurality of video frames corresponding to the one or more portions of the first digital video (Fig. 4, ¶39, “in response to a user selection to generate a timelapse video or a hyperlapse video. Regardless of the particular time-compressing technique used, the video compressing module 114 generates the time-compressed video 120 by sampling a subset of frames from the original video 118”, ¶84);
automatically smoothing a transition between each video frame of the plurality of video frames (¶56, “In general, the photometric stabilizer module 116 computes a weighted smoothing filter for each frame of the time-compressed video 120”),
wherein the second digital video depicts a timelapse sequence of the [video] (Fig. 4, ¶39, “in response to a user selection to generate a timelapse video or a hyperlapse video. Regardless of the particular time-compressing technique used, the video compressing module 114 generates the time-compressed video 120 by sampling a subset of frames from the original video 118”, ¶84, “Filters are computed for smoothing at least some photometric characteristic changes across the time-compressed video's frames”).
Harris-Ahmed and Bradley are analogous art to the claimed invention because they are from a similar field of endeavor of video processing, and automatically creating a seamless output video. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris to include the ability to use smooth a transition between each video frame resulting in resolutions as disclosed by Bradley with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris-Ahmed as described above to produce videos with the same quality as pre-planned cinemagraphs but without the painstaking effort and time consumption (Bradley, ¶3).
With regard to Claim 18,
Harris-Ahmed-Bradley teach the non-transitory computer-readable medium as described in claim 17, wherein the user input identifies the location by specifying the boundary within the digital content or a digital object displayed within the digital content in the user interface (Ahmed, Fig. 12, Fig. 24, 300, ¶¶64-65, “Screen Tag should be visible long enough for a user to notice it and to decide to whether or not to click and interact with it”, ¶77, “When clicking on the tag the user can access the additional information associated with that tag”).
Harris-Bradley and Ahmed are analogous art to the claimed invention because they are from a similar field of endeavor of interacting with displayed content to identify and search for related data. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Harris-Bradley to include the ability to identify the user input using a digital object displayed within the digital content in the user interface resulting in resolutions as disclosed by Ahmed with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Harris-Bradley as described above to provide users with a simple and fast form of searching and identifying editing data associated with a specific point of interest for the users which will increase users satisfaction by providing customized editing data that can be easily selected and for user’s previous editing operation based on the users’ interest to save the users’ time and effort. This is simply combining prior art elements according to known methods to yield predictable results, usage of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 19,
Harris-Ahmed-Bradley teach the non-transitory computer-readable medium as described in claim 17, the operations further comprising displaying the first : digital video and the second digital video in the user interface (Harris, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation. In other words, the system may be configured to allow a user to arbitrarily (and visually) navigate an undo/redo queue of the image editing application”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”).
The same motivation to combine for claim 17 equally applies for current claim.
With regard to Claim 20,
Harris-Ahmed-Bradley teach the non-transitory computer-readable medium as described in claim 19, wherein the displaying comprises displaying the second digital video in the user interface as the user input is detected (Harris, ¶43, “in response to receiving the indication of the invocation of the visual rewind operation, the method may include initiating the display of a sequence of frames (e.g., an animation) depicting the effect(s) of one or more image editing operations on an image during performance of the operation(s) in reverse order”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation. In other words, the system may be configured to allow a user to arbitrarily (and visually) navigate an undo/redo queue of the image editing application”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”).
The same motivation to combine for claim 17 equally applies for current claim.
Response to Arguments
Applicant argue that the current amendments overcome claims 1-16 rejection under 35 U.S.C. 101. Examiner respectfully disagrees, claims 1, 2-13, 15-16 is rejected under 35 U.S.C. 101 and refer the applicant to the detailed rejection for clarification. Examiner further notes that claims 17-20 are not rejected under 35 U.S.C. 101.
Applicant argue that Harris fails to teach "generating... [a] search query including a set of coordinates specifying a boundary of the area within the digital content," and "providing.. the search query as input to a search module," as recited in amended independent claim
Examiner respectfully disagrees, Harris disclose the ability to display the log data by displaying the edit steps in association with specific timepoints (video timeline) and location of editing (by showing the changes at specific points for specific frames and highlighting the edited area with the displayed frame) and the log data are searchable as the user is able to identify specific point of the log data (searching log data) by navigating to specific time point or frame See at least Fig. 1, Fig. 7, ¶4, “frame identifier may include a frame number and/or a timestamp, in various embodiments. The system may also be configured to capture data representing a log of interactions and corresponding changes in application state for the image editing application during performance of the one or more image editing operations”, ¶5, “system may be configured to determine a correlation between each of the entries in the log and a respective frame identifier. … each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶105, ¶113, ¶¶35-36, “frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps. In other embodiments, if the frame identifier of each frame includes a frame number, each entry of the log may include a corresponding frame number, rather than to a timestamp, and these frame numbers may be matched up to determine a correlation between frames of the animation and the log entries”, ¶98, “as the animation is built, the system may use information about the target location of each image editing operation to include hints about the region of interest for a given operation in the captured image data and/or interaction logs”, specific frame and portions of image is a location data within digital content. Ahmed further teach the same limitation by disclosing the ability to display added tags at specific locations within digital content the tags are searchable and could be selected by the user to show the associated data as annotations comments videos or links See at least Fig. 12, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 302, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”, adding a tag to a specific location is editing the displayed media content and selecting a tag that is associated with the annotation and displayed in the annotation location is a search for the location associated data that could include attached videos, Fig. 2, 13, Fig. 6-11, 13, ¶46, “computing device (14) features standard controls known to video streaming applications”, ¶56, “a play button, or other video control functions or buttons, are visible or are becoming visible (13) when a user clicks or interacts with the screen as shown in FIG. 6”, ¶49, the displayed tags associated with the selected frame is a search result.
Applicant argue that Harris fails to teach "searching... by: filtering out logs of the log data associated with location data indicating locations outside of the area; and obtaining, based on the filtering, a search result including logs of the log data associated with location data indicating locations disposed within the area," as recited in amended independent claim 1. Nowhere does Harris mention filtering-based searching, nor a search result based on locations within an area, as claimed.
Examiner respectfully disagrees, Harris teach the argued limitation by teaching the ability to display the edit operation associated with a frame with a time location searched (selected) by the user and excluding the edit operations associated with other frames(¶4, “frame identifier may include a frame number and/or a timestamp, ¶5, “each frame identifier includes a timestamp and each entry of the log includes a timestamp, determining a correlation may include identifying a log entry and a frame identifier that include matching timestamps”, ¶45, “system may be configured to allow a user to initiate a rewind operation, pause, fast forward, fast reverse, and/or rewind again in various combinations (causing various frames of the animation to be displayed in forward or reverse order) before identifying a point of interest in the animation”, ¶36, “the frame identifier of each frame of the animation includes a timestamp, and each entry of the log includes a timestamp, such a determination may be performed by searching the captured image data and interaction log for matching timestamps …”. Ahmed is relied on to modify Harris to include the ability the ability to search for edit operation associated with a specific area with a digital content See at least Fig. 12, Fig. 24, ¶84, “Screen Tags (20, 20 b, 20 c, 20 d, 20 e)”, Fig. 24, 300, ¶49, “user clicks on the side of the cowling of an airplane (20) to mark a location to place an annotation or Screen Tag. The Screen Tag serves as an anchor, placed by a user at a specific position (12) in a movie or image. This anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position”,.
In other words as clarified in the previous and current Office Actions Harris provide temporal search based on timestamps associated with frames . However, Ahmed teach the ability to display different tags that could be displayed on edited areas (adding comment or annotation or associating a file is editing) and are searchable as user can select to create a search query that return the data associated with the searched location (Fig. 12). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argue that the Screen Tags in Ahmed are metadata annotations applied to entire screens or frames, not detailed logs of edit operations with associated location data (see, e.g., Ahmed [0049]). Moreover, the Screen Tags in Ahmed operate on the video content itself, rather than on log data describing edit operations performed on the video content. The Office's suggestion that "adding a tag to a specific location is editing the displayed media content" stretches the meaning of an edit operation beyond what is reasonably supported by the claim language and specification.
Examiner respectfully disagrees,
First Ahmed tags is edits associated with a specific location or area with a digital content as shown in Fig. 4, Fig. 12 and ¶49, “anchor or Screen Tag (20) is associated with the (x,y) coordinate of the position (12) and the frame number. In another embodiment, the Screen Tag (20) may also be associated with time”.
Second, regarding the argument that Ahmed operate on the video content itself, rather than on log data describing edit operations performed on the video content. The current claim limitations does not distinguish between Ahmed and the invention as the video is a digital content and comment or attachments etc. are editing process for the digital content that the system represent using selectable tags at specific location that the user can select to search for the data associated with it (edit operation conducted at the tag specific location).
Third, the claims and the specification does not provide any explicit definition that for the edit operation. Therefore, under broadest reasonable interpretation any modification in the digital content is an editing process. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., what could be considered as “editing digital content”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant argue that a user simply clicking on a Screen Tag is not sufficient to teach or suggest "generating... [a] search query including a set of coordinates specifying a boundary of the area within the digital content; providing... the search query as input to a search module; [and] searching... the log data by: filtering out logs of the log data associated with location data indicating locations outside of the area," as recited in amended independent claim 1. Ahmed's system of selecting pre-existing tags does not involve generating a coordinate-based search query, nor searching log data via coordinate-based filtering.
Examiner respectfully disagrees, Ahmed explicitly clarify the usage of coordinates (x,y) within displayed content, and if the system does not identify the user click location and use it to identify the data associated with the selected location the automatically the system will not be functional as the user click location selecting a specific tag would not result in identifying the logged data associated with the selection location. In addition as the system identify the data associated with the selection it will automatically exclude the data associated with unselected location. In other words selecting a tag show related data and not any other unselected tags’ data.
Applicant argue Applicant respectfully submits that Ahmed's Screen Tags cannot simultaneously be all of these claimed features. The Office attempts to map a single feature of Ahmed's system to multiple distinct elements of the claimed invention. This interpretation fails to account for the specific technical implementation and interrelationships between these elements as described in claim 1. The claimed invention presents a cohesive system for generating, searching, and filtering detailed edit operation logs based on user- specified locations within digital content, a capability that is not suggested by Ahmed. Thus, Ahmed's Screen Tag system is incapable of performing "generating a search query based on the location, the search query including a set of coordinates specifying a boundary of the area within the digital content" and "searching, by the search module and based on the search query, the log data by: filtering out logs of the log data associated with location data indicating locations outside of the area; and obtaining, based on the filtering, a search result including logs of the log data associated with location data indicating locations disposed within the area," as recited in amended independent claim 1.
Examiner respectfully disagrees, Ahmed teach the argued limitation as Ahmed identify the user click location (search query based on location … area), based on that location Ahmed is able to search the logged data to identify the specific data associated with the selecting location and automatically exclude any other data that are associated with unselected tags (outside of the selection area) not associated with the selected tag (locations within selected area (pixels clicked)). Examiner notes that the claim does not limit the selected area in any specific form.
Applicant argue that claim 12 recite similar limitations to claim 1 and it is allowable for the same reasons provided earlier. In addition claim 12 disclose a rectangular selection area.
Examiner respectfully disagrees,
Claim 1 is not allowable and claim 12 is not allowable for similar reasoning. Regarding the argument related to the rectangular selection area, the arguments are moot as the current 35 USC 103 rejection of claim 12 does not rely on any of the previously presented references.
Applicant arguments related to Ayovan, the arguments are moot as the current 35 USC 103 rejection of claim 12 does not rely on Ayovan to teach the argued limitation.
Applicant submits that claim 17 is thus allowable for the reasons discussed above as Bradley does not cure the deficiencies of Harris.
Examiner respectfully disagrees,
Claim 1 is not allowable and claim 17 is not allowable for similar reasoning.
As to the remaining independent claims, applicant argue that they are allowable due to their respective direct and indirect dependencies upon one of the aforementioned Independent claims. The examiner respectfully disagrees, Independent claims were not allowable as stated in the paragraph above in this “Response to Arguments” section in this office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
US Patent No. 5786814 issued to Moran et al. that disclose a system for controlling the playback of a recorded session. Timestream and event information for the session is captured by one or more recording devices. Playback of a session is performed under the control of a session access device See at least Abstract.
US Patent Application Publication No. 2005/0034148 filed by Denny Jaeger that disclose the ability to provides a user with the ability to record his/her interactions with controls in the computer environment and then to replay these interactions See at least Abstract.
US Patent Application Publication No. 2016/0209989 filed by Ren et al. that disclose a system, non-transitory computer readable medium, and method for recording and replaying user operations See at least Abstract.
US Patent Application Publication No. 2017/0371898 filed by Sharma et al. that disclose the ability to create a search query based on a user input identifying a location within displayed content See at least Abstract, Fig. 1-3
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights