Apple Notes doesn't find certain substrings - Mac AND iOS

doogald

Trusted Member
Oct 23, 2012
2,771
109
63
Visit site
'iCloud engineering is working on it - as this is related to iCloud, not the OSs'
No further news.
I stated that when working offline, with no iCloud connection, the problem still exists.

Just guessing, but perhaps there is code sharing involved, so they fix on iCloud and that fix gets propagated to all OSes later.


I also stated that I will have to consider moving back to Evernote, as this problem gets increasingly annoying, and apparently no progress is being made (actually, I'm still running E in parallel, and whenever looking for something important, I cross-check in E, just in case there is information that Apple Notes search does not show me).

That's probably a smart plan.
 

Annie_M

Moderator
Mar 2, 2016
22,263
826
113
Visit site
Thanks for the update! I'm sorry no progress was made. Let's hope that the new year will bring better news! I am glad that you have a "backup" plan!
 

EdwinG

Ambassador
Mar 10, 2012
4,158
698
113
Visit site
It can be a very complex problem to fix.

First, just the software lifecycle is very elaborate and has multiple layers. One of the things is that they need to fix in their development environment, get that approved, then in staging, then approve. Then QA, then approve. And then finally it gets deployed to N servers. This can take a couple of months.

And then, it depends on how complicated and distributed (in other words, where else is it used) the code is. A simple fix can break other pieces elsewhere.

The fact that you have recurring calls with someone at Apple is a very promising sign :)
 

gnirkatto

Well-known member
Nov 4, 2017
570
6
18
Visit site
Yes I agree and appreciate Apple updating me on a regular basis, although no (visible) progress is being made.
I also understand this might take longer to fix, and that there are several QA steps to be taken before the regular public sees a result.
However, for me, the reliability of a notes app is critical. A few years ago, I gave up on using physical, written notebooks, in favor of a computer based app. This was Evernote in the beginning, as I came from the Blackberry world, then Android (BB Key 1/2), then iPhone. I kept Evernote, even on the iPhone, because I was used to it, because the Apple Notes app was not really feature rich (back then, a little better now), because I was too lazy to migrate.
Only after I made the decision to go fully Apple (also Mac, iPad, Watch), and only after the recent update of Apple Notes, which added a few nice features, I decided to migrate (which is quite easy when going E=>A, not so easy the other direction).
So for me, while understanding the challenges of a SW fixing & updating process, I have to look at it from a user's perspective: besides being safe, speedy & easy to use, it just HAS TO find EVERYTHING that I'm looking for, always. This is the main feature of a notes app.
I do have some patience, and will continue with both apps for a while, but sooner or later I will have to decide which way to go, finally.
 

gnirkatto

Well-known member
Nov 4, 2017
570
6
18
Visit site
P.S. when I discussed with the Apple support agent yesterday, about me considering going back to Evernote, she asked me to send this to the iCloud team as a feedback, together with the case number. I did that. She said that Apple is very interested in everybody using their stock apps (of course), so this kind of feedback might raise awareness. I think I said this before already: it's not easy to even become aware of this problem, so Apple may not have received a lot of complaints yet. Therefore the urgency might not be at the level that it should be.
 

gnirkatto

Well-known member
Nov 4, 2017
570
6
18
Visit site
Actually, I just wanted to post a final update, as I had an update call with Apple support 2 days ago. Here it is:

on a support call a few weeks ago, I was told that 'a substring will only be found if it reappears as a full & standalone word somewhere else'
I told them that this is incorrect, because

assuming a string ABCDEFG
searching DEFG => not found
searching CDEFG => found
while CDEFG never ever shows anywhere else in this or in other notes
So they told me they will reconsider this.

2 days ago/last friday:

Apple confirmed that it currently 'works as designed'. Some substrings may not be found.
They also told that they now have it on their list to 'improve search algorithms over time'.

The case is now closed, but still unresolved. Of course there was no timeframe given for a solution (this I never expected, nobody does that).
So all we can do now is to check from time to time, and hope for a resolution sometime (or use a different notes app instead).

But at least, it's supposedly 'on their list' now.
 

gnirkatto

Well-known member
Nov 4, 2017
570
6
18
Visit site
I'm going to continue this, as it's so striking. Sorry it's gonna be long and complicated.
This post refers to Apple pictures (on the Mac), no longer to Notes, but the issue with search is very similar to the one experienced in Notes. A moderator may want to edit the title to Notes & Pictures.

Anticipated conclusion: search in Pictures is broken (as in Notes). Or designed badly.

New picture in the library. Its title field is 'Schiff in Venedig' (German for ship in Venice. I'll continue with the German terms in case someone wants to reproduce exactly).
No tags, keywords, location, people contained nor added. The AI algorithm did NOT identify a 'Schiff' in 3 days (but that's OK and not the point of this post - but important to understand there was no index for 'Schiff' created for this particular picture).

I performed the following searches:

'Schiff' ==> not found (but 158 other pictures containing a 'Schiff' were found - identified by AI, so I assume contained in the index of the library)
'Venedig' ==> found (I don't have 'Venedig' in any other picture in my library. No tag, no title, nothing)
'Schiff ' (trailing blank) ==> not found, but 'Schiff in Venedig' proposed for search
'Schiff in' ==> found, and proposed

Then I continued and created a tag 'Schiff' and assigned it to the picture from Venedig. After indexing complete:

'Schiff' ==> 159 pictures (now including the one from Venedig) found instantly
Tag 'Schiff' deleted.
'Schiff' ==> 158 pictures only
So why does it find 'Venedig' and not 'Schiff' - as both are in the title field only???

Theory: if it doesn't find a search word in the index (containing tags & AI identified), it continues to search in the title fields. As I have many 'Schiff's in the Index, but no 'Venedig', it stops after finding 158 'Schiff's - but goes straight to the title field, after not finding 'Venedig' in the Index.

To further investigate, I created a tag 'Venedig' and assigned it to a different picture (NOT to the one with 'Ship'.

The search for:

'Venedig' ==> the newly tagged picture found instantly - but no longer the one with 'Schiff' (but it was found before the tag 'Venedig was created - see above)
'Vened' ==> both pictures found !!! So the STANDALONE term 'Vened' was not in the index, therefore continued to search in the title field (but as it was found as a substring of an indexed word, why not also show it (maybe that's how it works))
'Venedig ' (trailing blank) ==> found the newly tagged only, but displays the Venedig/Schiff picture in search proposal.

I think that proves the following:

when search word found in index - stops, no further search
when search word not in index - continues search in other fields
when search word found as substring in index - displays picture & continues search in other fields

But: apparently, in all thinkable search situations, it may show a proper search proposal - which is good, however, there can be MANY proposals, and the small window does not show EVERYTHING. And, results are shown only for words BEGINNIG with the search string (eg 'Vened' - yes, 'enedi' - no),so it's easy to overlook things.

So maybe this 'works as designed'? However, I consider that a severe bug, as having to keep all these 'rules' in mind and always obeying all of them, alternatively not being sure to find all elements one is looking for, is totally unacceptable imho.

This is highly complex, and I'm still not sure that I have looked into all possible variants and made the right conclusions. But shouldn't this be a very basic and totally easy to use and to understand feature of every app/database that has a purpose in life, to quickly and reliably find all elements that a user looks for?

I tried to send Apple feedback on my findings via the feedback site, but am not sure I was able to explain the case properly, due to its complexity. And no, I do not want to go through the regular support structures again, as this is a complete PITA once you already reached the level of insight into a problem like I did (I think), and until you get to a person who is smart an wants to understand the issue.

So honestly, I don't know what to do about that, other than writing about it here.
 

Trees

Well-known member
Dec 26, 2012
1,173
11
38
Visit site
Great writeup and information. Good to know that indexing and search may be enhanced in the future.

If iOS is using the locate command (or iOS equivalent) behind the scenes to search the database (mlocate.db), then if the database has not been refreshed/updated to reflect new file names or attributes (e.g. the strings you have added to the file/photo metadata); the search for the new string value(s) may not be found.

The links below are examples that have more details about the locate command and database. As noted in the updatedb article, the database is refreshed on a schedule, perhaps daily.

I could be completely wrong here on the idea that iOS is using something like locate and mlocate.db.

Ubuntu Manpage: locate - find files by name
Ubuntu Manpage: updatedb - update a database for mlocate
 

gnirkatto

Well-known member
Nov 4, 2017
570
6
18
Visit site
I don't think the problem lies with the DB not (yet) being refreshed.
1. the behavior is the same, even after several days of waiting (for a potential automatic refresh to complete)
2. everything that I described above was AFTER indexing was complete, after having added new strings, tags etc. This can be seen when typing something randomly into the search field - it will say 'indexing' underneath a list of proposed searches. Once completed, 'indexing' goes away. My library consists of ~24k pictures, and indexing after adding new tags etc takes between 30 and 90 seconds (on the M1 Pro Mac).
 

doogald

Trusted Member
Oct 23, 2012
2,771
109
63
Visit site
well - this is the problem, and explains why not many people immediately run into this issue.
It seems to be particular combinations of characters & words, that work, or not.
Normally, when starting to work with such an app, nobody will do extensive testing - nor is it possible - to figure out if EVERY combination works. This happens mainly by accident, like in my case.
All of the not working strings have been in my notes app since forever, AND I rebuilt the search index according to Apples advice, with no success.

In the meanwhile, 3 Apple people, independent from each other, could recreate the problem on their computers (and on mine). Yesterday we recorded a demo on my computer, to be sent to Apple R & D.

I did not want to bother you with the original string-combination that made problems, because it contains language specific special characters. But I found another note titled "Question for the Avengers Endgame specialist", making the same problems:

Avengers : found
venger : not found
vengers : found
avenger : found
endgame : not found (nor any substring version)
specialist : found
cialist : not found
pecialist : found

It might also be that the same strings that don't work on mine, do work on other computers, due to problems in the indexing process, but this is a thing only R&D people will be able to clarify, imho.

Fwiw, all but “cialist” are found when I search on ios16 and Ventura.
 

gnirkatto

Well-known member
Nov 4, 2017
570
6
18
Visit site
I believe that silently, Apple fixed this, at least in Notes.
All searches that didn't work (as described in the beginning of this thread), work now.
 

Trending Posts

Members online

Forum statistics

Threads
260,764
Messages
1,768,212
Members
441,265
Latest member
vber