DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicant
Limitations appearing inside of {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Claims 1-17, 19 and 20 are pending in the application.
This office action is in response to application filed on 09/26/2023. Amended claims filed on 09/26/2023 are considered in the following office action. Claims 1, 4, 8, 9, 10, 12, and 17 are amended. Claims 19 and 20 are new. Claim 18 is canceled. No new matter has been introduced.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17, 19 and 20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea (Step 2A Prong 1) without additional imitations that integrate the abstract idea into a practical application (Step 2A Prong 2) and without amounting to significantly more than the abstract idea (Step 2B).
Independent Claim 1 recites a method. Thus the claim is directed to a process which is a statutory category of invention (MPEP §2106.03). (Step 1: YES)
Step 2A Prong 1 evaluates if the claim recites any judicial exception (MPEP §2106.04(a)). The claim recites: A method comprising:
obtaining a digital image of a board having a plurality of individual notes, wherein each of the individual notes represents a physical note;
separating the notes in the digital image;
converting handwriting in the notes to corresponding character strings;
detecting a color for each of the notes;
creating color-coded groups of the separated notes based upon the detected colors;
outputting the color-coded groups with the corresponding character strings.
The USPTO has enumerated groupings of abstract ideas (See §MPEP 2106.04(a)(2)), defined as:
I) Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
II) Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
III) Mental processes – concepts performed in the human mind with or without physical aid (i.e., pen and paper, or by using a computer). These include an observation, evaluation, judgment, or opinion.
The limitations [B], [C], [D], and [E] each recite a mental process which is recognized by the court as an abstract idea (See MPEP §2106.04(a)(2)). Regarding limitations [B], [D], and [E], separating notes in the digital image, detecting color and creating groups are acts that may be performed in the human mind. Regarding limitations [C] and [F] are mental process that can be performed pen and paper. The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim (MPEP §2106.04(a)(2)). Additionally, separating notes in a “digital image” is merely reciting the words “apply it” (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP § 2106.05(f)). (Step 2A Prong 1: YES)
Step 2A Prong 2 analysis evaluates weather the claim recites additional elements that integrate the exception in to a practical application of that exception according to MPEP §2106.04(d) by: 1) Identify additional elements recited in the claim beyond the judicial exception; and 2) If additional elements are identified, then evaluate the additional elements both individually and in combination to determine whether the claim as a whole integrates the exception into a practical application.
Additional elements include [A] which is merely data gathering contributes only nominally or insignificantly to the execution of the claimed method (e.g., in a data gathering step or in a field-of-use limitation) would not integrate a judicial exception or provide significantly more (MPEP §,2106.05(g)). (Step 2A Prong 2: NO)
Step 2B analysis evaluates if the claim as a whole amounts to significantly more than the judicial exception. The additional elements, as stated in Prong 2, do not amount to significantly more than the judicial exception. An ‘improvement cannot be to the judicial exception itself, i.e., ‘apply it’/automating an otherwise mentally/manually performed process, even if requiring the use of a computer as a tool (MPEP §2106.05(a)). These elements are found not to be enough to qualify as "significantly more" (MPEP 2106.05(d)) (Step 2B: NO).
Claims 19 and 20 are similarly analyzed as claim 1. The addition of processors, memory and non-transitory storage medium merely add performing a mental process on a generic computer (MPEP §2106.05(a)(I)), see Specification page 2, paragraph 3, shown below.
PNG
media_image1.png
410
862
media_image1.png
Greyscale
Claims 2-9, 11, 13-15 and 17 merely add details on the mental processes of note recognition and grouping of the notes which is an abstract idea that may be performed in the mind with the use of physical aid (MPEP §2106.04(a)(2)), without adding significantly more than the abstract idea.
Claim 10 and 17merely adds details to the notes (i.e., adding a numerical value, the note is a task) which are insignificant extra solution activity (MPEP §2106.05(g)).
Claim 16 merely adds details indicating a plotter which is insignificant extra solution activity (MPEP §2106.05(g)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 13-15, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Biller et al., US 20140294236 A1, hereinafter “Biller”, in view of Antony et al., US 20190108202 A1, hereinafter Antony.
Regarding claim 1, Biller teaches A method comprising:
obtaining a digital image of a board having a plurality of individual notes, wherein each of the individual notes represents a physical note (See Fig 3A exhibits capture a visual representation of a plurality of notes 310A, and the notes are physical notes and it is more efficient to capture more than one note at a time; Biller, ¶[0046]);
separating the notes in the digital image (See Fig 3A exhibits Recognize one of the plurality of notes from the visual representation 320A; Biller, ¶[0046]);
{converting handwriting} in the notes to corresponding character strings (The system extracts content of the one of the plurality of notes (step 330A). … the system can apply image transformation to at least part of the visual representation before extracting content (i.e., converting); ¶[0046]; physical notes can include hand-written Post-It.RTM. notes, paper, or film, white-board with drawings, posters, and signs; Biller, ¶[0022]);
detecting a color for each of the notes (a note recognition/management system may use multiple recognition algorithms to recognize notes and extract notes' content, such as color recognition; Biller, ¶[0053]; The shape and/or color of the note itself may be used as the mark to facilitate the identification, recognition, and/or authentication of a note; Biller, ¶[0039]);
creating color-coded groups of the separated notes based upon the detected colors (processing unit 110 can label a note based on its specific shape, color, content, and/or other information of the note. For example, each group of note can have a different color (e.g., red, green, yellow, etc.); Biller, ¶[0042]); and
outputting the color-coded groups with the corresponding character strings (the system may present the extracted content of the plurality of notes with an output field indicating the categories of the notes; Biller, ¶[0055]).
Biller does not explicitly disclose converting handwriting in the notes to corresponding character strings.
However, Antony, a similar field of endeavor of digital content management system for creating a digital document from handwritten content, teaches converting handwriting in the notes to corresponding character strings (digital content management system recognizes handwritten text characters within the digital image and thereby converts the handwritten text into digital text by creating corresponding digital text portions; Antony, ¶[0069]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include converting handwritten text as taught by Antony to the invention of Biller. The motivation to do so would be to enable an individual to later view handwritten notes captured in a digital image, incorporate the handwritten notes into other electronic documents, and/or collaborate with other users using the handwritten notes.
Claim 19 is similarly analyzed as analogous claim 1. Biller further teaches A computing device comprising: at least one memory; and at least one processor coupled to at least one of the at least one memory, the at least one processor being configured to perform operations (mobile device 15 may include one or more processors, microprocessors, internal memory and/or data storage and other electronic circuitry for executing software or firmware to provide the functionality described herein; Biller, ¶[0025]).
Claim 20 is similarly analyzed as analogous claim 1. Biller further teaches at least one non-transitory computer-readable memory encoded with instructions that, when executed, configure at least one processor (mobile device 15 includes one or more programmable processors 70 configured to operate according to executable instructions (i.e., program code), typically stored in a computer-readable medium or data storage 68; Biller, ¶[0032]).
Regarding claim 2, the combination of Biller and Antony teaches the method of claim 1. Biller further teaches further comprising separating the notes from a background in the image (the system may use color spaces such as the RGB, HSV, CIELAB, etc. to identify regions of interest corresponding to the notes for color recognition; Biller, ¶[0053]; marks 520 are much brighter than the notes 510A and the background; ¶[0050]).
Regarding claim 3, the combination of Biller and Antony teaches the method of claim 2. Biller further teaches wherein the separating the notes from the background comprises converting the digital image into a hue/saturation/value format (the system may use color spaces such as the RGB, HSV, CIELAB, etc. to identify regions of interest corresponding to the notes for color recognition; Biller, ¶[0053]; marks 520 are much brighter than the notes 510A and the background; ¶[0050]).
Regarding claim 4, the combination of Biller and Antony teaches the method of claim 1. Biller further teaches wherein separating further comprises segmenting the notes (recognition and segmentation of notes on a visual representation produced by an image sensor; Biller, ¶[0054]).
Regarding claim 5, the combination of Biller and Antony teaches the method of claim 4. Biller further teaches further comprising using contour detection to find a border of each of the notes (the mobile device 15 is configured to recognize note(s) by determining the general boundary of the note(s); Biller, ¶[0028]).
Regarding claim 6, the combination of Biller and Antony teaches the method of claim 5. Biller further teaches comprising fitting rectangles to the border of each of the notes (identify regions of interest corresponding to the notes for color recognition; Biller, ¶[0053]).
Regarding claim 13, the combination of Biller and Antony teaches the method of claim 1. Biller further teaches wherein each of the notes has markings indicating a location of handwriting to be converted (the processing unit 110 can authenticate a note before extracting the content of the note. If the note is authenticated, the content will be extracted and stored in the note content repository 140. In some cases, the processing unit can extract the authentication information from the mark on the note; Biller, ¶[0040]).
Regarding claim 14, the combination of Biller and Antony teaches the method of claim 13. Biller further teaches wherein the markings comprise lines in a corner of the notes (each note 510A has a mark 520 has two elements--two retroreflective rectangular tags at upper left and lower bottom corners; Biller, ¶[0050]).
Regarding claim 15, the combination of Biller and Antony teaches the method of claim 13. Biller further teaches wherein the markings comprise one or more logos in a corner of the notes (the mark on a note can include a symbol, for example, a company logo, a matrix code, a barcode, a color code, or the like; Biller, ¶[0040]).
Claims 7-9, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Moore et al., US 20150106755 A, hereinafter “Moore ‘755”.
Regarding claim 7, the combination of Biller and Antony teaches the method of claim 6. Biller further teaches further comprising applying {masking} to each of the rectangles to hide a content of each of the notes except a particular area of the notes (the notes are further distinguished in their shape and due to the presence of unique patterns detected by shape recognition (e.g., Hough transform, shape context, etc.) and pattern recognition algorithms (e.g., Support Vector Machine, cross-correlation, template matching, etc.) respectively. These algorithms help filter out unwanted objects in the visual representation or other sources of notes' content and leave only those regions of interest corresponding to the notes; Biller, ¶[0053]). While Biller teaches algorithms that define regions of interest in the note, Biller does not explicitly disclose applying masking.
However, Moore ‘755, in a similar field of endeavor of note content capturing, recognition, extraction, and/or management tools and systems, teaches applying masking to each of the rectangles to hide a content of each of the notes except a particular area of the notes (determine a mask that indicates which pixels of digital note 746 are background pixels and which pixels of digital note 746 are foreground pixels; Moore ‘755, ¶[0134]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include masks as taught by Moore ‘755 to the combined invention of Biller and Antony. The motivation to do so would be to indicate background or foreground pixels.
Regarding claim 8, the combination of Biller, Antony and Moore ‘755 teaches the method of claim 7. Antony further teaches wherein converting further comprises converting the handwriting from only the particular area of the notes (digital content management system 104 identifies handwritten text portions and/or handdrawn figure portions (e.g., pictures, graphs, spreadsheets, etc.), Antony, ¶[0069]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include converting handwriting from only certain areas as taught by Antony to the combined invention of Biller and Moore. The motivation to do so would be to differentiates and processes each content portion based on the type of content portion (e.g., digital content management system 104 processes a text portion different from a figure portion)..
Regarding claim 9, the combination of Biller and Antony discloses the method of claim 1. Biller does not explicitly disclose wherein detecting further comprises detecting the color based upon a median hue value for each of the notes.
However, Moore ‘755 teaches wherein detecting further comprises detecting the color based upon a median hue value for each of the notes (note enhancement module 89 may select a pixel value for the uniform background color as an average or median color value of pixels included in the background region of the digital note; Moore ‘755, ¶[0126]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include background color based on median hue as taught by Moore ‘755 to the combined invention of Biller and Antony. The motivation to do so would be to determine a uniform background color for the digital note.
Regarding claim 17, the combination of Biller and Antony teaches the method of claim 1. Biller does not explicitly disclose wherein the converting step further comprises converting the handwriting representing task information.
However, Moore ‘755 teaches wherein the converting step further comprises converting the handwriting representing task information (the notes may represent tasks; Moore ‘755, ¶[0109]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include handwriting representing tasks as taught by Moore ‘755 to the combined invention of Biller and Antony. The motivation to do so would be to provide management assistance to situations where people participate in a collaboration session by writing information, such as tasks, on paper-based notes, such as Post-It.RTM.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Suntinger et al., US 10083412 B2, as cited in the IDS.
Regarding claim 10, the combination of Biller and Antony teaches the method of claim 1. Biller does not explicitly disclose wherein converting further comprises converting the character strings into point values for the corresponding notes.
However, Suntinger, a similar field of endeavor of scheduling work items that form part of a project, teaches wherein converting further comprises converting the character strings into point values for the corresponding notes (Each assignment defines an activity of the work item, a resource (assigned to work on the activity), an iteration (in which the resource works on the activity), a resource commitment (e.g. a number of story points); Suntinger, [Col 13:46-54]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include converting character strings into point values as taught by Suntinger to the combined invention of Biller and Antony. The motivation to do so would be to capture an expected measure or resource/skill to be spent spend on the activity.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Moore, US 10846634 B1, hereinafter Moore ‘634.
Regarding claim 11, the combination of Biller and Antony teaches the method of claim 1. Biller does not explicitly teach further comprising generating charts based upon the color-coded groups and the corresponding character strings.
However, Moore ‘634, a similar field of endeavor of task organization planning systems to share information about progress and scheduling of tasks among multiple individuals, teaches further comprising generating charts based upon the color-coded groups and the corresponding character strings (These date assignments will then be synchronized back to the website where detailed Gantt charts, reports, comparisons, histograms, etc. can be produced; Moore ‘634, [Col 12:38:41]; This graph 500 will include a line for each “Responsible”, and will be colored by “Responsible.”; [Col 16:14-15])]; This will be a stacked graph 505 and will use a different color for each “reason.”; [Col 16: 27-28]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include generating charts as taught by Moore ‘634 to the combined invention of Biller and Antony. The motivation to do so would be for ease of review of progress or failures.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Kudssi, US 10810265 B2.
Regarding claim 12, the combination of Biller and Antony teaches the method of claim 1. Biller does not explicitly disclose wherein obtaining further comprises obtaining an image of a Scrum board. However, Kudssi teaches wherein obtaining further comprises obtaining an image of a Scrum board (A digital image of the Scrum board 100 is captured 404 such that each notecard 112 is visible in the image; Kudssi; [Col 4:28-30]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include imaging a Scrum board as taught by Kudssi to the combined invention of Biller and Antony. The motivation to do so would be to acquire information from physical notes captured on a white board during stand-up meetings, facilitating Agile Scrum development process.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Avida, US 6116707 A.
Regarding claim 16, the combination of Biller and Antony teaches the method of claim 1. Biller does not explicitly teach further comprising sending the color-coded groups with the corresponding character strings to a robotic plotter for use in re-creating the board.
However, Avida, a similar field of endeavor of output applications when it is desired to place a recording medium, such as paper or a whiteboard, teaches further comprising sending the color-coded groups with the corresponding character strings to a robotic plotter for use in re-creating the board (incoming signal enters presentation signal reception device 138, which splits the image signal 12 from an optional sound signal, and sends the image signal 12, typically a digital signal, to the plotter transmitter module 14; Avida, [Col 6:51-57]; plotter transmitter module 14 … transmits signals 16 to the robotic plotter printhead 18; [Col 6:4-7]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include sending notes to a robot plotter arm to re-re-create the board as taught by Avida to the combined invention of Biller and Antony. The motivation to do so would be to automate the replication of the information on a vertical or highly include surface.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANDHANA PEDAPATI whose telephone number is 571-272-5325. The examiner can normally be reached M-F 8:30am-6pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANDHANA PEDAPATI/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669