Error when running SMS module on mvt-android

[bordenc]

On a clean install, ran an ADB backup (which has apparently been deprecated, is there a plan to deal with that?), and got the following output (scrubbed of identifying information):

MVT - Mobile Verification Toolkit
                https://mvt.re
                Version: 2.6.1
                Indicators updates checked recently, next automatic check in 12 hours


         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_AmnestyTech_investigations_master_2021-07-18_nso_pegasus.stix2  
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_mvt-project_mvt-indicators_main_intellexa_predator_predator.stix
                  2                                                                                                                                  
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_mvt-project_mvt-indicators_main_2022-06-23_rcs_lab_rcs.stix2    
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_AssoEchap_stalkerware-indicators_master_generated_stalkerware.st
                  ix2                                                                                                                                
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_AmnestyTech_investigations_master_2023-03-29_android_campaign_ma
                  lware.stix2                                                                                                                        
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_mvt-project_mvt-indicators_main_2023-04-11_quadream_kingspawn.st
                  ix2                                                                                                                                
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_mvt-project_mvt-indicators_main_2023-06_01_operation_triangulati
                  on_operation_triangulation.stix2                                                                                                   
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_mvt-project_mvt-indicators_main_2023-07-25_wyrmspy_dragonegg_wyr
                  mspy_dragonegg.stix2                                                                                                               
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_AmnestyTech_investigations_master_2024-05-02_wintego_helios_wint
                  ego_helios.stix2                                                                                                                   
         INFO     [mvt.android.cmd_check_backup] Parsing STIX2 indicators file at path                                                               
                  /home/.local/share/mvt/indicators/raw.githubusercontent.com_AmnestyTech_investigations_master_2024-12-16_serbia_novispy_novi
                  spy.stix2                                                                                                                          
         INFO     [mvt.android.cmd_check_backup] Loaded a total of 10722 unique indicators                                                           
         INFO     [mvt] Checking Android backup at path: backup.ab                                                                                   
         INFO     [mvt.android.modules.backup.sms] Running module SMS...                                                                             
         INFO     [mvt.android.modules.backup.sms] Processing SMS backup file at apps/com.android.providers.telephony/d_f/000000_sms_backup          
(truncated for brevity)       
         INFO     [mvt.android.modules.backup.sms] Processing MMS backup file at apps/com.android.providers.telephony/d_f/000036_mms_backup          
         ERROR    [mvt.android.modules.backup.sms] Error in running extraction from module SMS: cannot access local variable 'message_links' where it
                  is not associated with a value                                                                                                     
                  Traceback (most recent call last):                                                                                                 
                    File "/home/.local/pipx/venvs/mvt/lib/python3.13/site-packages/mvt/common/module.py", line 171, in run_module             
                      exec_or_profile("module.run()", globals(), locals())                                                                           
                      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                           
                    File "/home/.local/pipx/venvs/mvt/lib/python3.13/site-packages/mvt/common/utils.py", line 263, in exec_or_profile         
                      exec(module, globals, locals)                                                                                                  
                      ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                  
                    File "<string>", line 1, in <module>                                                                                             
                    File "/home/.local/pipx/venvs/mvt/lib/python3.13/site-packages/mvt/android/modules/backup/sms.py", line 61, in run        
                      self.results.extend(parse_sms_file(data))                                                                                      
                                          ~~~~~~~~~~~~~~^^^^^^                                                                                       
                    File "/home/.local/pipx/venvs/mvt/lib/python3.13/site-packages/mvt/android/parsers/backup.py", line 241, in parse_sms_file
                      if message_links or entry["body"].strip() == "":                                                                               
                         ^^^^^^^^^^^^^                                                                                                               
                  UnboundLocalError: cannot access local variable 'message_links' where it is not associated with a value                            
         INFO     [mvt.android.cmd_check_backup] Reference hash of the info.json file:                                                               
                  "71330ec54f4675c37b00263d4f3b43a3b2822569a8dfed0b4d9f1e4f7b899a78"                                                                 
         INFO      NOTE: Using MVT with public indicators of compromise (IOCs) WILL NOT automatically detect advanced attacks.                       
                                                                                                                                                     
                  Please seek reputable expert help if you have serious concerns about a possible spyware attack. Such support is available to human 
                  rights defenders and civil society through Amnesty International's Security Lab at https://securitylab.amnesty.org/get-help/?c=mvt

No idea what to do with this. Suggestions?

[besendorf]

Hi,
thanks for reporting that. Unfortunatly I am not able to reproduce this. In my test backup there is no MMS database. That might be the problem. Would you be willing to share your MMS database with me?

Regarding adb backup being deprecated. That is the case for a few years now already. Unfortunately Google/Android has not come up with something else and it is the only manufacturer independent way to access sms history for example. So for now we are sticking with it.

[bordenc]

Would you be willing to share your MMS database with me?

Not to impeach your discretion or trustworthiness, but I hope you sympathise that I have many personal messages that I don't feel comfortable realising into the wild. I doubt my contacts would approve, either. Could I build a "sandboxed" or "dummy" archive for you to test? Alternatively, if there's a reliable way to backup, wipe and restore my messages, I might be able to generate a usable file for you.

Unfortunately Google/Android has not come up with something else and it is the only manufacturer independent way to access sms history for example. So for now we are sticking with it.

So much for "not being evil," Google. I understand.

[bordenc]

I think I can run mvt through a debugger to the exception. What should I look for (other than obvious null variables)?

[besendorf]

Not needed so far. I would try to push a fix later today that you can test on your backup.

[besendorf]

Just pushed a fix to main. Could you try again please?

git pull
pip install .

[bordenc]

That seems to have fixed that problem. Well done! Tests pass on a local backup. When going over an ADB bridge, I have two issues:

1. The timeout is too short. If I don't approve ADB access quickly enough, the mvt-android command times out.
2. If the command hangs on a test, CTRL+C doesn't seem to break out of it. I have to kill the terminal. However, they all eventually finish.

But, otherwise, the command works (and my phone looks clean). Thank you, again!

[besendorf]

Great.

1. Timeout is currently 5s do you think that's too short? Also it should retry after those 5s. Code is here: https://github.com/mvt-project/mvt/blob/main/src/mvt/android/modules/adb/base.py
2. which modules excatly is that?

btw we recommend not checking backups but rather using check-adb as it includes much more data

[bordenc]

I'm very grateful for your prompt responses to my posts. My semi-solicited, unprofessional, free thoughts below:

1. Timeout is currently 5s do you think that's too short? Also it should retry after those 5s. Code is here: https://github.com/mvt-project/mvt/blob/main/src/mvt/android/modules/adb/base.py

It'll sound nuts, but I'd start the timeout at a minute. At least on my device, I sometimes have to fight with the lock screen, enter backup passwords, or just notice that the process stalled because it was waiting for me to reauthorise ADB access (that's more a quirk with me, I prefer to be notified every time ADB accesses my phone than just set it to trust my computer). You know the API whereas I don't, but I don't think a minute timeout would increase the overall runtime. It would allow more retries before the process failed.

What's definitely not happening is a process reattempt. It exits with Unable to connect to the device over USB. Try to unplug, plug the device and start again. if it times out. It also does that if the device goes to sleep - but I suspect that's an ADB issue, well beyond your control.

2. which modules excatly is that?

I re-ran the module from scratch, and it pauses the longest on mvt.android.modules.adb.packages for a few minutes. Other pauses are a few seconds, so OK. I apologise for not taking accurate measurements, which I can probably do with a bit more planning. Is there such a thing as inserting process timers into a debugger?

These pauses are perfectly justified given the work that the program is doing. It's just on a first run-through or two, a new (impatient) user like me might not know whether the program got caught in an infinite loop. It's UI aesthetics, not an actual bug.

I haven't been able to recreate the CTRL+C hang on the git clone, but I'll comment if I run into it again, which I probably won't now that I'm actually being patient with the program :-)