DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicant
Limitations appearing inside of {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Claims 1-5, 9, 11-17, and 19-22 are pending in the application.
Response to Amendments
The Amendment filled 07/09/2026 in response to Non-Final Office Action mailed 02/10/2026 has been entered.
Claims 1, 5, 17, 19, and 20 have been amended.
Claims 6-8, 10, and 18 have been canceled.
Claims 21 and 22 are newly added.
Rejections of 02/10/2026 under 35 USC §§101 and 103 have been withdrawn in light of amended claims.
Response to Arguments/Remarks
Applicant’s arguments, see Remarks, filed 07/09/2026, with respect to claims 1-17, and 19-20 have been fully considered and are persuasive. The 35 USC §101 Rejection of 02/10/2026 has been withdrawn.
Applicant’s arguments with respect to claims 1-17, 19-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
In response to applicant's argument (“applying Avida to Briller would result in printing a single flat photograph of the original board, not re-creating the board”, See Remarks, filed on 07/09/2026, page 9), a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim.
Information Disclosure Statement
No Information Disclosure Statement (IDS) was filed; therefore, no applicant-submitted references were considered.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-5, 9, 11-17, 19-22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1, 19 and 20, the claims recite two instances of converting handwriting into character strings, one from notes and one from only the particular area of the notes. It is unclear which character string is referred to in the last line of the claim reciting “the character string”.
Claims 2-5, 9, 11-17, and 21-22 are rejected because they depend from rejected claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 9, 13-15, 17, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over “Biller” (Biller et al., US 20140294236 A1), in view of “Antony” (Antony et al., US 20190108202 A1), in view of “Almutairi” (Almutairi, Abdullah, and Meshal Almashan. "Instance segmentation of newspaper elements using mask R-CNN." 2019 18th IEEE International conference on machine learning and applications (ICMLA). IEEE, 2019.), and further in view of Suntinger et al., US 10083412 B2, as cited in the IDS.
Regarding claim 1, Biller teaches a method comprising:
obtaining a digital image of a board having a plurality of individual notes, wherein each of the individual notes represents a physical note (See Fig 3A exhibits capture a visual representation of a plurality of notes 310A, and the notes are physical notes and it is more efficient to capture more than one note at a time; Biller, ¶[0046]);
separating the notes in the digital image (See Fig 3A exhibits Recognize one of the plurality of notes from the visual representation 320A; Biller, ¶[0046]);
{fitting rectangles to a border of each of the notes};
{applying masking to each of the rectangles to hide content of each of the notes except a particular area of the notes};
{converting handwriting in the notes to corresponding character strings};
{converting handwriting from only the particular area of the notes into corresponding character strings representing point values for corresponding notes};
detecting a color for each of the notes (a note recognition/management system may use multiple recognition algorithms to recognize notes and extract notes' content, such as color recognition; Biller, ¶[0053]; The shape and/or color of the note itself may be used as the mark to facilitate the identification, recognition, and/or authentication of a note; Biller, ¶[0039]);
creating color-coded groups of the separated notes based upon the detected colors (processing unit 110 can label a note based on its specific shape, color, content, and/or other information of the note. For example, each group of note can have a different color (e.g., red, green, yellow, etc.); Biller, ¶[0042]); and
outputting the color-coded groups with the corresponding character strings(the system may present the extracted content of the plurality of notes with an output field indicating the categories of the notes; Biller, ¶[0055]).
Biller does not explicitly disclose
fitting rectangles to a border of each of the notes;
applying masking to each of the rectangles to hide content of each of the notes except a particular area of the notes;
converting handwriting in the notes to corresponding character strings;
converting handwriting from only the particular area of the notes into corresponding character strings representing point values for corresponding notes;
However, Almutairi, a similar field of endeavor of physical document digitization, teaches fitting rectangles to a border of each of the notes (providing images with a bounding box; Almutairi, [page 1372, §III. A., col 1, ¶1]);
applying masking to each of the rectangles to hide content of each of the notes except a particular area of the notes and a segmentation mask,…, to find the regions that most likely contain elements; Almutairi, [§III. A., p1372, col 1, ¶1]; and see Fig 1, shown below, exhibits applying masks of each of the element rectangles.
PNG
media_image1.png
265
459
media_image1.png
Greyscale
);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include bounding box and segmentation as taught by Almutairi to the invention of Biller . The motivation to do so would be for article segmentation and classification of different main elements of the document (e.g., articles, advertisements, and page headers)..
However, Antony, a similar field of endeavor of digital content management system for creating a digital document from handwritten content, teaches converting handwriting in the notes to corresponding character strings (digital content management system recognizes handwritten text characters within the digital image and thereby converts the handwritten text into digital text by creating corresponding digital text portions; Antony, ¶[0069]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include converting handwritten text as taught by Antony to the invention of Biller. The motivation to do so would be to enable an individual to later view handwritten notes captured in a digital image, incorporate the handwritten notes into other electronic documents, and/or collaborate with other users using the handwritten notes.
Antony teaches converting handwriting from only the particular area of the notes into corresponding character strings {representing point values for corresponding notes}, as shown above in claim 1.
The combination of Biller, Almutairi, and Antony does not explicitly disclose representing point values for corresponding notes.
However, Suntinger teaches corresponding character strings representing point values for corresponding notes (Each assignment defines an activity of the work item, a resource (assigned to work on the activity), an iteration (in which the resource works on the activity), a resource commitment (e.g. a number of story points); Suntinger, [Col 13:46-54]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include converting character strings into point values as taught by Suntinger to the combined invention of Biller, Antony and Almutairi. The motivation to do so would be to capture an expected measure or resource/skill to be spent spend on the activity.
Claim 19 is similarly analyzed as analogous claim 1. Biller further teaches a computing device comprising: at least one memory; and at least one processor coupled to at least one of the at least one memory, the at least one processor being configured to perform operations (mobile device 15 includes one or more programmable processors 70 configured to operate according to executable instructions (i.e., program code), typically stored in a computer-readable medium or data storage 68; Biller, ¶[0032]).
Claim 20 is similarly analyzed as analogous claim 1. Biller further teaches at least one non-transitory computer-readable memory encoded with instructions that, when executed, configure at least one processor (mobile device 15 includes one or more programmable processors 70 configured to operate according to executable instructions (i.e., program code), typically stored in a computer-readable medium or data storage 68; Biller, ¶[0032]).
Regarding claim 2, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller further teaches further comprising separating the notes from a background in the image (the system may use color spaces such as the RGB, HSV, CIELAB, etc. to identify regions of interest corresponding to the notes for color recognition; Biller, ¶[0053]; marks 520 are much brighter than the notes 510A and the background; ¶[0050]).
Regarding claim 3, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 2. Biller further teaches wherein the separating the notes from the background comprises converting the digital image into a hue/saturation/value format (the system may use color spaces such as the RGB, HSV, CIELAB, etc. to identify regions of interest corresponding to the notes for color recognition; Biller, ¶[0053]; marks 520 are much brighter than the notes 510A and the background; ¶[0050]).
Regarding claim 4, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller further teaches wherein separating further comprises segmenting the notes (recognition and segmentation of notes on a visual representation produced by an image sensor; Biller, ¶[0054]).
Regarding claim 5, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller further teaches wherein the boarder of each of the notes is found using contour detection (the mobile device 15 is configured to recognize note(s) by determining the general boundary of the note(s); Biller, ¶[0028]).
Regarding claim 9, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller further teaches wherein detecting further comprises detecting the color based upon a median hue value for each of the notes (the system may use color spaces such as the RGB, HSV, CIELAB, etc. to identify regions of interest corresponding to the notes for color recognition; Biller, ¶[0053]).
Regarding claim 13, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller further teaches wherein each of the notes has markings indicating a location of handwriting to be converted (the processing unit 110 can authenticate a note before extracting the content of the note. If the note is authenticated, the content will be extracted and stored in the note content repository 140. In some cases, the processing unit can extract the authentication information from the mark on the note; Biller, ¶[0040]).
Regarding claim 14, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 13. Biller further teaches wherein the markings comprise lines in a corner of the notes (each note 510A has a mark 520 has two elements--two retroreflective rectangular tags at upper left and lower bottom corners; Biller, ¶[0050]).
Regarding claim 15, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 13. Biller further teaches wherein the markings comprise one or more logos in a corner of the notes (the mark on a note can include a symbol, for example, a company logo, a matrix code, a barcode, a color code, or the like; Biller, ¶[0040]).
Regarding claim 17, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1.
Antony further teaches wherein the method further comprises a second converting step comprising converting second handwriting on the notes representing task information (creating the digital content portion for each of the one or more content portions can include converting handwritten text within the one or more content portions identified within the digital image into digital text; Antony, ¶[0131]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to converting text withing one or more content portions on the notes as taught by Antony to the combined invention of Biller and Almutairi and Suntinger. The motivation to do so would be to recognize specialized text such as computer code, hyperlinks, task lists, etc.
Claims 11 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Almutairi, and in view of Suntinger, and further in view of “Moore ‘634” (Moore, US 10846634 B1).
Regarding claim 11, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller does not explicitly teach further comprising generating charts based upon the color-coded groups and the corresponding character strings.
However, Moore ‘634, a similar field of endeavor of task organization planning systems to share information about progress and scheduling of tasks among multiple individuals, teaches further comprising generating charts based upon the color-coded groups and the corresponding character strings (These date assignments will then be synchronized back to the website where detailed Gantt charts, reports, comparisons, histograms, etc. can be produced; Moore ‘634, [Col 12:38:41]; This graph 500 will include a line for each “Responsible”, and will be colored by “Responsible.”; [Col 16:14-15])]; This will be a stacked graph 505 and will use a different color for each “reason.”; [Col 16: 27-28]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include generating charts as taught by Moore ‘634 to the combined invention of Biller, Antony, Almutairi, and Suntinger. The motivation to do so would be for ease of review of progress or failures.
Regarding claim 22, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller further teaches wherein obtaining the digital image comprises capturing multiple video frames (the note management system may receive a visual representation (e.g., a video) of a number of notes taken by a video recording device and a text stream of notes that is entered via a laptop. In some embodiments, the multiple sources are sources of notes' content taken from different devices, for example, cameras, scanners, computers, etc; Biller, ¶[0055]), and
{wherein converting the handwriting comprises creating composite images of the particular area over the multiple video frames to find a best estimate of the point values}.
The combination does not explicitly disclose wherein converting the handwriting comprises creating composite images of the particular area over the multiple video frames to find a best estimate of the point values.
However, Moore ‘634, in addition to teaching wherein obtaining the digital image comprises capturing multiple video frames (the camera may include video capability or the capability to collect multiple images in rapid succession; Moore, ‘634, [Col 9: 20-23]),
further teaches and wherein converting the handwriting comprises creating composite images of the particular area over the multiple video frames to find a best estimate of the point values (A given digital image 70 may be captured as a single image, or it may be a composite of multiple shots of the board 2; Moore,’634, [Col 7:23-25]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include composite image from multiple shots, or video frames as taught by Moore ‘634 to the combined invention of Biller, Antony, Alutairi, and Suntinger. The motivation to do so would be to allow the user to scan the entire board 2, if the board 2 (or the portion of the board 2 to be analyzed) is too large in one or more dimensions to be captured within the field of a single photograph.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, and further in view of Kudssi, US 10810265 B2.
Regarding claim 12, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. Biller does not explicitly disclose wherein obtaining further comprises obtaining an image of a Scrum board. However, Kudssi teaches wherein obtaining further comprises obtaining an image of a Scrum board (A digital image of the Scrum board 100 is captured 404 such that each notecard 112 is visible in the image; Kudssi; [Col 4:28-30]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include imaging a Scrum board as taught by Kudssi to the combined invention of Biller, Antony, Almutairi, and Suntinger. The motivation to do so would be to acquire information from physical notes captured on a white board during stand-up meetings, facilitating Agile Scrum development process.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Biller in view of Antony, in view of Almutairi, in view of Suntinger, and further in view of Avida, US 6116707 A.
Regarding claim 16, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 1. The combination does not explicitly teach further comprising sending the color-coded groups with the corresponding character strings to a robotic plotter for use in re-creating the board.
However, Avida, a similar field of endeavor of output applications when it is desired to place a recording medium, such as paper or a whiteboard, teaches further comprising sending the color-coded groups with the corresponding character strings to a robotic plotter for use in re-creating the board (incoming signal enters presentation signal reception device 138, which splits the image signal 12 from an optional sound signal, and sends the image signal 12, typically a digital signal, to the plotter transmitter module 14; Avida, [Col 6:51-57]; plotter transmitter module 14 … transmits signals 16 to the robotic plotter printhead 18; [Col 6:4-7]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include sending notes to a robot plotter arm to re-re-create the board as taught by Avida to the combined invention of combination of Biller, Antony, Almutairi, and Suntinger. The motivation to do so would be to automate the replication of the information on a vertical or highly include surface.
Claim 21 is rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Biller in view of Antony, in view of Almutairi, in view of Suntinger, and further in view of "Tseng", (Tseng et al., US 20210144110 A1).
Regarding claim 21, the combination of Biller, Antony, Almutairi, and Suntinger teaches the method of claim 4. The combination does not explicitly disclose wherein segmenting the notes comprises applying a watershed algorithm to separate the individual notes that overlap on the board.
However, Tseng, a similar field of endeavor of textual-content extraction from digital images, teaches wherein segmenting the notes comprises applying a watershed algorithm to separate the individual notes that overlap on the board (the present invention provides a text-detection method that combines the U-Net segmentation with Watershed algorithm—the latter is known for segmenting mutually touching objects; Tseng, ¶[0062]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include watershed algorithm as taught by Tseng to the combined invention of Biller, Antony, Almutairi, and Suntinger. The motivation to do so would be to obtain an optimal pixelated text-distribution map free of overlapping text.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Valtech (Woods, H., Green, T. (2019, October 30). Digitising sticky notes with Deep Learning. URL: https://www.valtech.com/blog/sticky-note-detection/.) teaches the digitization of sticky notes from digital images and would have been relied upon for teaching obtaining digital image of a board with physical notes, fitting rectangles to each note, converting handwriting, detecting color, and creating color-coded groups.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANDHANA PEDAPATI whose telephone number is 571-272-5325. The examiner can normally be reached M-F 8:30am-6pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANDHANA PEDAPATI/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights